task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-pcxjp-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-pcxjp-kubeconfig [get-kubeconfig] Found admin password secret: cluster-pcxjp-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-pcxjp-password [get-kubeconfig] API Server URL: https://a06caf0690c83408e8e1ab76b1cacb04-6bd3147b9bc78608.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.561d7479-5fb9-4171-86d4-486c1d7a884e.prod.konfluxeaas.com [clone-repo] sync-master-to-release-workflow [clone-repo] https://github.com/pierDipi/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 74% (2305/3110) Updating files: 75% (2333/3110) Updating files: 76% (2364/3110) Updating files: 77% (2395/3110) Updating files: 78% (2426/3110) Updating files: 79% (2457/3110) Updating files: 80% (2488/3110) Updating files: 81% (2520/3110) Updating files: 82% (2551/3110) Updating files: 83% (2582/3110) Updating files: 84% (2613/3110) Updating files: 85% (2644/3110) Updating files: 86% (2675/3110) Updating files: 87% (2706/3110) Updating files: 88% (2737/3110) Updating files: 89% (2768/3110) Updating files: 90% (2799/3110) Updating files: 91% (2831/3110) Updating files: 92% (2862/3110) Updating files: 93% (2893/3110) Updating files: 94% (2924/3110) Updating files: 95% (2955/3110) Updating files: 96% (2986/3110) Updating files: 97% (3017/3110) Updating files: 98% (3048/3110) Updating files: 99% (3079/3110) Updating files: 100% (3110/3110) Updating files: 100% (3110/3110), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:2e623de9a93ccce79950e917affe03d05d0a7eff69143d7481fdb39990c764f1 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:2e623de9a93ccce79950e917affe03d05d0a7eff69143d7481fdb39990c764f1 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:f46055ca7ed2a087dc4bd22965ed61ae14027df0373b324f95f4d6998b378a9e [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:f46055ca7ed2a087dc4bd22965ed61ae14027df0373b324f95f4d6998b378a9e [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:f94a36664a6553b816dc04c45e472a62cdcdb731a5f667f29ab4b59a2331a34e [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:f94a36664a6553b816dc04c45e472a62cdcdb731a5f667f29ab4b59a2331a34e [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0baee72a16ecd2affc3d37ff066dd732a21a630131a2071be8ca82e011a3405c [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0baee72a16ecd2affc3d37ff066dd732a21a630131a2071be8ca82e011a3405c [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.24) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1714 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1714 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1714 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.84s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 302ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 46ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a06caf0690c83408e8e1ab76b1cacb04-6bd3147b9bc78608.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (25/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (55/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-8fdjb 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-8fdjb condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-n9q2x 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-n9q2x condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-6tscq 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-6tscq condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:f46055ca7ed2a087dc4bd22965ed61ae14027df0373b324f95f4d6998b378a9e [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0baee72a16ecd2affc3d37ff066dd732a21a630131a2071be8ca82e011a3405c [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:2e623de9a93ccce79950e917affe03d05d0a7eff69143d7481fdb39990c764f1 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:f94a36664a6553b816dc04c45e472a62cdcdb731a5f667f29ab4b59a2331a34e [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-7787c55b6-zt6xb 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-7787c55b6-zt6xb condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.24) [e2e-llm-inference-service] 🎯 Server version (4.21.24) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-tlsc4 1/1 Running 0 14s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-tlsc4 condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-p9b99 1/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-p9b99 condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-866675cbb8-cvzqk 1/1 Running 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-tsj82 1/1 Running 0 80s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-tsj82 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-tsj82 1/1 Running 0 90s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-tsj82 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-6b8548bc6d-wmkwl" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-6b8548bc6d-7b946 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.24 [e2e-llm-inference-service] Kubernetes Version: v1.34.9 [e2e-llm-inference-service] ClusterVersion desired: 4.21.24 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.24 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:f46055ca7ed2a087dc4bd22965ed61ae14027df0373b324f95f4d6998b378a9e [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:163c2ed5cd155bb2dd26cd9e0dc7cc66f1367771344f0c3498e8235eea4e42c8 [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:5ce69c3ae5bc6b3b4888343ec0c55c7e1c8a05304b25bfbd157358fc4c26ee52 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0baee72a16ecd2affc3d37ff066dd732a21a630131a2071be8ca82e011a3405c [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:0baee72a16ecd2affc3d37ff066dd732a21a630131a2071be8ca82e011a3405c [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'metadata': '3.1.1', 'asyncio': '0.23.8', 'json-report': '1.5.0', 'anyio': '4.9.0', 'xdist': '3.6.1', 'httpx': '0.30.0', 'cov': '5.0.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: metadata-3.1.1, asyncio-0.23.8, json-report-1.5.0, anyio-4.9.0, xdist-3.6.1, httpx-0.30.0, cov-5.0.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-09 17:52:38.382 6468 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-09 17:52:38.382 6465 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-09 17:52:38.439 6465 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 17:52:38.439 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 17:52:38.475 6465 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 17:52:38.475 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 17:52:38.488 6465 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 17:52:38.488 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-09 17:52:46.142 6465 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 17:52:46.142 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 17:52:46.178 6465 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 17:52:46.178 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-09 18:02:17.423 6465 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:02:17.423 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:02:17.473 6465 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:02:17.473 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-09 18:09:14.508 6465 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:09:14.508 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:09:14.547 6465 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:09:14.547 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 18:09:14.547 6465 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:09:14.547 6465 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:09:14.551 6465 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:09:14.551 6465 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:09:14.562 6465 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:09:14.562 6465 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 18:09:14.563 6465 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:09:14.563 6465 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:09:14.566 6465 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 18:09:14.566 6465 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 18:09:14.575 6465 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 18:09:14.575 6465 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-09 18:20:51.135 6465 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:20:51.135 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:20:51.175 6465 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:20:51.175 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:20:51.182 6465 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:20:51.182 6465 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 18:20:51.182 6465 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:20:51.182 6465 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:20:51.185 6465 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:20:51.185 6465 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:20:51.193 6465 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:20:51.193 6465 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 18:20:51.193 6465 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:20:51.193 6465 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 18:20:51.197 6465 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 18:20:51.197 6465 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 18:20:51.206 6465 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 18:20:51.206 6465 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_n... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:816: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'workload-sin... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-09T18:05:05.639263', start_time = 1783620305.6395905 [e2e-llm-inference-service] duration = 12.227129697799683, timestamp_end = '2026-07-09T18:05:17.866722' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_n... {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900 [e2e-llm-inference-service] extra_headers = {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'} [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] > test_case.response_assertion(response) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1120: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] response = [e2e-llm-inference-service] [e2e-llm-inference-service] def response_assertion(response: requests.Response) -> None: [e2e-llm-inference-service] assert response.status_code == 200, ( [e2e-llm-inference-service] f"Expected 200, got {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] body = response.json() [e2e-llm-inference-service] data = body.get("data", []) [e2e-llm-inference-service] assert data, f"Expected non-empty data[], got: {response.text}" [e2e-llm-inference-service] ids = [m.get("id") for m in data] [e2e-llm-inference-service] for model_id in model_ids: [e2e-llm-inference-service] > assert model_id in ids, ( [e2e-llm-inference-service] f"Expected model {model_id!r} in data[].id, found: {ids}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Expected model 'lora-adapter-1' in data[].id, found: ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] E assert 'lora-adapter-1' in ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:120: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-09T18:03:39.827864] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_name='llmisvc-model-fb-opt-125m-with-ba4d693a', endpoint='/v1/models', max_tokens=20, payload_formatter=None, response_assertion=.response_assertion at 0x7f77edef07c0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-09T18:03:39.840708] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-09T18:03:39.944587] end - ✅ in 0.104s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-09T18:03:39.944727] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Inference Pool kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-09T18:04:06Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: "False" (reason "BackendNotFound", message "backend(llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local) not found")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-09T18:04:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:04:07Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:04:07Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-09T18:04:07Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T18:04:06Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T18:04:07Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T18:04:18Z', 'status': 'True', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T18:04:18Z', 'severity': 'Info', 'status': 'True', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T18:03:45Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-09T18:05:05.639029] end - ✅ in 85.694s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-09T18:05:05.639263] start - args=(, TestCase(base_refs=['router-managed', 'workload-single-cpu', 'model-fb-opt-125m-with-lora-hf'], prompt=None, service_name='llmisvc-model-fb-opt-125m-with-ba4d693a', endpoint='/v1/models', max_tokens=20, payload_formatter=None, response_assertion=.response_assertion at 0x7f77edef07c0>, wait_timeout=900, response_timeout=60, extra_headers={'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}, url_getter=, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': {'X-Gateway-Model-Name': 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'}} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_model_routing_url] [2026-07-09T18:05:05.639599] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'llmisvc-model-fb-opt-125m-with-ba4d693a', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-llmisvc-model-fb-66e80b02'}, [e2e-llm-inference-service] {'name': 'workload-single-cpu-llmisvc-mod-1ae2b31a'}, [e2e-llm-inference-service] {'name': 'model-fb-opt-125m-with-lora-hf-c0d503b0'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:181 Found model-routing URL for llmisvc-model-fb-opt-125m-with-ba4d693a: http://acb4c897ca637469f965f729e7820bc9-1887918949.us-east-1.elb.amazonaws.com/ (name='gateway-external-model-routing', path='/') [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_model_routing_url] [2026-07-09T18:05:05.653179] end - ✅ in 0.013s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://acb4c897ca637469f965f729e7820bc9-1887918949.us-east-1.elb.amazonaws.com/v1/models with payload None [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 200: {"object":"list","data":[{"id":"facebook/opt-125m","object":"model","created":1783620317,"owned_by":"vllm","root":"/mnt/models","parent":null,"max_model_len":2048,"permission":[{"id":"modelperm-a52440a50925c408","object":"model_permission","created":1783620317,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]},{"id":"publishers/kserve-ci-e2e-test/mo [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-09T18:05:17.866722] end - ❌ 12.227s: Expected model 'lora-adapter-1' in data[].id, found: ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] assert 'lora-adapter-1' in ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf] ❌ ERROR: Failed to call llm inference service llmisvc-model-fb-opt-125m-with-ba4d693a: Expected model 'lora-adapter-1' in data[].id, found: ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] assert 'lora-adapter-1' in ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'llmisvc-model-fb-opt-125m-with-ba4d693a' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:39Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:39Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:39Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '34173' [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-llmisvc-model-fb-66e80b02 [e2e-llm-inference-service] - name: workload-single-cpu-llmisvc-mod-1ae2b31a [e2e-llm-inference-service] - name: model-fb-opt-125m-with-lora-hf-c0d503b0 [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://acb4c897ca637469f965f729e7820bc9-1887918949.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://acb4c897ca637469f965f729e7820bc9-1887918949.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:07Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:06Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://acb4c897ca637469f965f729e7820bc9-1887918949.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-5ff8d867d8-w25vd to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:56 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:56 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:56 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:42 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:57 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:12 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.33:8000/health": dial tcp 10.134.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-5ff8d867d8-w25vd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:53 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:53 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:53 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:54 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:54 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:54 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:57 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:57 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-565d6d8849 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-5ff8d867d8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:51 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-7d99555666-77kds to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:50 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:17 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 26.831s (26.831s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:17 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:17 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:55:56 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.30:8000/health": dial tcp 10.134.0.30:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:32 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:47 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.30:8000/health": dial tcp 10.134.0.30:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-7d99555666-77kds [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:49 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:49 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:49 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:32 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:32 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-6f9bd5659b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-7d99555666 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:15 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-6c6c746446-f2ppq to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:23 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:23 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:23 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:10 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.32:8000/health": dial tcp 10.134.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:21 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:37 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.32:8000/health": dial tcp 10.134.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-6c6c746446-f2ppq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4 to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:21 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:21 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:21 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:21 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:21 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-6c6c746446 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:59:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5wqrb5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5wqrb5 to ip-10-0-141-118.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:45 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.335s (1.335s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:45 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:45 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:14 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bf6wbv to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:32 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:21 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:21 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:21 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:05:07 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.36:8000/health": dial tcp 10.134.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46-c2vcs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-fbc5f5d46 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:05:17 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2 to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:51 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:27 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:40 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:40 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:40 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:45 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:34 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495-vthvl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:36 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:36 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:36 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:37 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:37 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:37 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:20 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.133.0.38:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-7f5c854898 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-f445c495 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:56 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:48 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:48 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:48 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:04:54 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.134.0.37:8000/health": dial tcp 10.134.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-684fcd8f47 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:46 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:46 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:46 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:42 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:04:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:04:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:05:04 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:51 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddhqdr7 to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:42 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:29 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:12 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:23 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:58 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:00 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6 to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:55:26 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:50 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:55:26 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:55:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schepcjxs to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:08 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:19 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 11.153s (11.153s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:19 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:19 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:51 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:00:51 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schem88rf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schem88rf to ip-10-0-141-118.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:51 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:53 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.769s (1.769s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:53 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:53 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:55:26 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:55:26 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-66ddf487d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:17 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:03:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7 to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:56 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:56 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:56 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:42 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:57:42 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetvb8m [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schetvb8m to ip-10-0-141-118.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:26 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:12 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:01:12 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlxsp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlxsp to ip-10-0-141-118.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:56:13 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:00 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:58:00 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-gjgcf to ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:15 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" in 3.683s (3.683s including waiting). Image size: 301879232 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:15 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:15 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:19 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.022s (1.022s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:20 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:24 kserve-ci-e2e-test kubelet/ip-10-0-132-16.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:16 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:18 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 2.521s (2.521s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:19 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:24 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-56cbd96798-zfl5f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-56cbd96798-gjgcf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:12 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:12 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:12 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:13 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:14 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.548s (1.548s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:15 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 33.527s (33.527s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:02 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.133.0.29:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:24 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:24 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:24 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-64c46df48 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-56cbd96798 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:01 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:23 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:54:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:48 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-s6w2m to ip-10-0-141-118.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:23 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:25 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.506s (2.506s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:44 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:44 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:43 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.132.0.47:15021/healthz/ready": read tcp 10.132.0.2:52664->10.132.0.47:15021: read: connection reset by peer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:44 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:50 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.47:15021/healthz/ready": dial tcp 10.132.0.47:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-rs72c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-54c47cdfc7-rs72c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-s6w2m [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:58 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:58 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:58 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:03 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.133.0.27:15021/healthz/ready": dial tcp 10.133.0.27:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:28 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:35 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Warning Unhealthy Readiness probe failed: Get "http://10.133.0.27:15021/healthz/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn to ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:39 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:43 kserve-ci-e2e-test kubelet/ip-10-0-141-44.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-rg72r to ip-10-0-141-118.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:18 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:18 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:18 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:20 kserve-ci-e2e-test kubelet/ip-10-0-141-118.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:38 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-bswgn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-bswgn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-9zb2v [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-9zb2v [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-799f46c59b-rg72r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:18 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-799f46c59b-rg72r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:17 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-799f46c59b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:38 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:53:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:18 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-799f46c59b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:42 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 17:52:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-54c47cdfc7 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:17 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 18:02:51 kserve-ci-e2e-test GatewayAuthBootstrap Warning ReconcileError Failed to delete gateway PodMonitor: failed to delete PodMonitor: podmonitors.monitoring.coreos.com "router-gateway-1-metrics" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-07-09 18:03:44.620 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models'), ('hf://edbeeching/opt-125m-lora', '/mnt/lora/lora-adapter-1')] [e2e-llm-inference-service] 2026-07-09 18:03:44.620 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-07-09 18:03:47.634 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-09 18:03:47.635 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 3.014127998000049 seconds. [e2e-llm-inference-service] 2026-07-09 18:03:47.635 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://edbeeching/opt-125m-lora to local [e2e-llm-inference-service] 2026-07-09 18:03:48.038 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://edbeeching/opt-125m-lora to /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] 2026-07-09 18:03:48.038 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.4038290649998544 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 (EngineCore pid=70) Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 2.74it/s] [e2e-llm-inference-service] (EngineCore pid=70) [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:04:58 [model_executor/model_loader/default_loader.py:384] Loading weights took 0.37 seconds [e2e-llm-inference-service] (EngineCore pid=70) WARNING 07-09 18:04:58 [model_executor/layers/utils.py:282] Failed to create oneDNN linear, fallback to torch linear. Exception: could not create a primitive descriptor for the matmul primitive. Run workload with environment variable ONEDNN_VERBOSE=all to get additional diagnostic information. [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:04:58 [lora/punica_wrapper/punica_selector.py:20] Using PunicaWrapperCPU. [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:04:58 [v1/core/kv_cache_utils.py:1319] GPU KV cache size: 29,056 tokens [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:04:58 [v1/core/kv_cache_utils.py:1324] Maximum concurrency for 2,048 tokens per request: 14.19x [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:04:59 [v1/worker/cpu_model_runner.py:82] Warming up model for the compilation... [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:04:59 [v1/worker/gpu_model_runner.py:5342] ubatch_slices: None, ubatch_slices_padded: None [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:04:59 [lora/model_manager.py:878] Adding lora. Model id: 1, int id: 1 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:00 [lora/model_manager.py:58] Removing adapter int id: 1 [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:05:00 [v1/worker/cpu_model_runner.py:92] Warming up done. [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:05:00 [v1/engine/core.py:283] init engine (profile, create kv cache, warmup model) took 1.31 seconds [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:00 [tokenizers/registry.py:68] Loading CachedHfTokenizer for tokenizer_mode='hf' [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [utils/gc_utils.py:40] GC Debug Config. enabled:False,top_objects:-1 [e2e-llm-inference-service] (EngineCore pid=70) INFO 07-09 18:05:01 [config/vllm.py:790] Asynchronous scheduling is disabled. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [v1/engine/utils.py:1158] READY from local core engine process 0. [e2e-llm-inference-service] (EngineCore pid=70) WARNING 07-09 18:05:01 [config/vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [v1/metrics/loggers.py:273] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 227 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/openai/api_server.py:590] Supported tasks: ['generate'] [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:878] Adding lora. Model id: 1, int id: 1 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:285] Activating LoRA. int id: 1, slot index: 0 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.0.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.0.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.0.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.0.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.1.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.1.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.1.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.1.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.2.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.2.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.2.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.2.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.3.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.3.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.3.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.3.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.4.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.4.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.4.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.4.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.5.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.5.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.5.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.5.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.6.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.6.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.6.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.6.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.7.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.7.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.7.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.7.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.8.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.8.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.8.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.8.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.9.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.9.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.9.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.9.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.10.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.10.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.10.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.10.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.11.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.11.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.11.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.11.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/.../models/serving.py:201] Loaded new LoRA adapter: name 'lora-adapter-1', path '/mnt/lora/lora-adapter-1' [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:58] Removing adapter int id: 1 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:58] Removing adapter int id: 1 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:878] Adding lora. Model id: 2, int id: 2 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:285] Activating LoRA. int id: 2, slot index: 0 [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.0.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.0.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.0.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.0.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.1.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.1.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.1.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.1.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.2.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.2.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.2.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.2.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.3.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.3.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.3.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.3.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.4.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.4.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.4.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.4.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.5.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.5.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.5.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.5.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.6.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.6.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.6.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.6.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.7.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.7.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.7.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.7.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.8.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.8.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.8.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.8.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.9.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.9.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.9.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.9.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.10.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.10.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.10.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.10.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:303] Successfully loaded LoRA weights for module model.decoder.layers.11.self_attn.qkv_proj. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.11.self_attn.out_proj, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.11.fc1, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [lora/model_manager.py:293] No LoRA weights found for module model.decoder.layers.11.fc2, skipping. [e2e-llm-inference-service] (EngineCore pid=70) DEBUG 07-09 18:05:01 [v1/engine/core.py:1158] EngineCore waiting for work. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/.../models/serving.py:201] Loaded new LoRA adapter: name 'publishers/kserve-ci-e2e-test/models/lora-adapter-1', path '/mnt/lora/lora-adapter-1' [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/base.py:197] Warming up chat template processing... [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:141] There is no chat template fallback for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [renderers/hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] Failed to load AutoTokenizer chat template for /mnt/models [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] Traceback (most recent call last): [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/vllm/renderers/hf.py", line 120, in resolve_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] return tokenizer.get_chat_template(chat_template, tools=tools) [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1825, in get_chat_template [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] raise ValueError( [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/hf.py:122] ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:01 [renderers/base.py:205] This model does not support chat template. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/openai/api_server.py:594] Starting vLLM server on https://0.0.0.0:8000 [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:37] Available routes are: [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /openapi.json, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /docs, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /redoc, Methods: GET, HEAD [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /tokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /detokenize, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /load, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /version, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /health, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /metrics, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/models, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /ping, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /ping, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /invocations, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/chat/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/responses, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/completions, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/messages, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /inference/v1/generate, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /scale_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/chat/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:01 [entrypoints/launcher.py:46] Route: /v1/completions/render, Methods: POST [e2e-llm-inference-service] (APIServer pid=1) INFO: Started server process [1] [e2e-llm-inference-service] (APIServer pid=1) INFO: Waiting for application startup. [e2e-llm-inference-service] (APIServer pid=1) INFO 07-09 18:05:02 [entrypoints/ssl.py:60] SSLCertRefresher monitors files: ['/var/run/kserve/tls/tls.key', '/var/run/kserve/tls/tls.crt'] [e2e-llm-inference-service] (APIServer pid=1) INFO: Application startup complete. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:04 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:04 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:05 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:06 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:07 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:08 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:09 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:10 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:11 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:12 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:12 [v1/metrics/loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:13 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:14 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:14 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:15 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:16 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:17 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] (APIServer pid=1) DEBUG 07-09 18:05:18 [v1/engine/async_llm.py:875] Called check_health. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'storage-initializer' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 2026-07-09 18:03:44.864 1 storage.initializer INFO [initializer-entrypoint:():17] Initializing, args: (src_uri, dest_path): [('hf://facebook/opt-125m', '/mnt/models')] [e2e-llm-inference-service] 2026-07-09 18:03:44.864 1 storage.initializer INFO [kserve_storage.py:download():166] Copying contents of hf://facebook/opt-125m to local [e2e-llm-inference-service] 2026-07-09 18:03:44.864 1 storage.initializer INFO [kserve_storage.py:download():169] Allow patterns: ['tokenizer.json', 'tokenizer_config.json', 'special_tokens_map.json', 'vocab.json', 'merges.txt', 'config.json', 'generation_config.json'] [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/8_PA_wEVGiVa2goH2H4KQOQpvVY=.b3fb716a3024261980becb2382e31a3780985130.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_0877d851-0e59-4159-95a2-f58a1a9b97d2'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/3EVKVggOldJcKSsGjSdoUCN1AyQ=.cf739e3ba86db7791ebab2828cc34b8a5acd3a86.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_29f9bd75-7265-47d3-8ac6-699452a576c4'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_fd39ec41-0260-44d2-80d4-23ad749794d3'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/ahkChHUJFxEmOdq5GDFEmerRzCY=.5dfa36546b8eddce0e04df3133c30df43fcc3828.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_c1060521-dda8-4bed-aaf1-1dcab5a7da0c'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/vzaExXFZNBay89bvlQv-ZcI6BTg=.27c24ca9d908d0b678b20c698aeb9e950c44d865.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_58a845ac-e849-42e5-854c-dbf3bfd8cc8e'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] Could not set the permissions on the file '/mnt/models/.cache/huggingface/download/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.0a39732b2d8be8e493cab3da68b68cc3e28221de.incomplete'. Error: [Errno 13] Permission denied: '/mnt/tmp_976ffcac-d3e0-4cbb-8111-a839d0061863'. [e2e-llm-inference-service] Continuing without setting permissions. [e2e-llm-inference-service] 2026-07-09 18:03:45.259 1 storage.initializer INFO [kserve_storage.py:download():234] Successfully copied hf://facebook/opt-125m to /mnt/models [e2e-llm-inference-service] 2026-07-09 18:03:45.259 1 storage.initializer INFO [kserve_storage.py:download():235] Model downloaded in 0.39457200500010003 seconds. [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1783620226.331256,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1783620226.3313622,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n- type: max-score-picker\n- type: lora-affinity-scorer\nschedulingProfiles:\n- name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n weight: 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3314764,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3318968,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3319335,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: single-profile-handler} {Type: queue-scorer} {Type: prefix-cache-scorer} {Type: max-score-picker} {Type: lora-affinity-scorer}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: lora-affinity-scorer, Weight: 4.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.331945,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3321884,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"single-profile-handler","parameters":null,"type":"single-profile-handler"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"},{"name":"lora-affinity-scorer","parameters":null,"type":"lora-affinity-scorer"}],"schedulingProfiles":[{"name":"default","plugins":[{"pluginRef":"lora-affinity-scorer","weight":4},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1783620226.347012,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3471262,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3471608,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3471973,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3472567,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3472757,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3473265,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: single-profile-handler, Type: single-profile-handler} {Name: queue-scorer, Type: queue-scorer} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: lora-affinity-scorer, Type: lora-affinity-scorer} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: lora-affinity-scorer, Weight: 4.00} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3473756,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1783620226.347436,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.347459,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.347488,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"approx-prefix-cache-producer"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3475583,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3475673,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [lora-affinity-scorer/lora-affinity-scorer: 4.000000, queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000032840 DataConfig:{Sources:[{Plugin:0xc0004bc750 Extractors:[0xc000032a40]}]} FlowControlConfig: ParserRegistry:0xc000033100}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3628936,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629284,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629465,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629537,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629603,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629675,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629742,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629816,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629885,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3629963,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3630037,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3630104,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.363025,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.363032,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: single-profile-handler/single-profile-handler, Profiles: map[default:{Filters: [], Scorers: [lora-affinity-scorer/lora-affinity-scorer: 4.000000, queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3630633,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3630724,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3630912,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3631058,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1783620226.363132,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3632367,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1783620226.363251,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3632848,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3635778,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3636699,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3637395,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3639042,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3639388,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3639534,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.363965,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3650806,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.365157,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1783620226.367772,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3677955,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3679245,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.3689067,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.465207,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.4652286,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783620226.465229,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.465247,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783620226.465248,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.4652607,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783620226.4653492,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","reconcileID":"f415c747-8d3d-44ff-93de-cd621d5e8be6"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.5653844,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1783620226.5654035,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1783620245.7437153,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool","reconcileID":"e6f8c9f2-e28b-4c43-9db7-3e7127d20e32"} [e2e-llm-inference-service] {"level":"info","ts":1783620304.2733319,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv","reconcileID":"7f646b19-5a52-493b-8dd8-c4f82899c824"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f430f847-cf88-4025-9b76-efd3d27792c8 [e2e-llm-inference-service] resourceVersion: '33375' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] nodeName: ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr [e2e-llm-inference-service] uid: 6c5e7685-a76b-42fa-8048-fb682f709bfe [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 70819630-8ff6-4231-9b71-e8335ee8d820 [e2e-llm-inference-service] resourceVersion: '34158' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.134.0.37 [e2e-llm-inference-service] nodeName: ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv [e2e-llm-inference-service] uid: ce9d7d7f-5c2b-4845-a794-a578803d5d9a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: ce9d7d7f-5c2b-4845-a794-a578803d5d9a [e2e-llm-inference-service] resourceVersion: '34156' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 78595fb58d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.37/23"],"mac_address":"0a:58:0a:86:00:25","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.37/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.37\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:25\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d [e2e-llm-inference-service] uid: 3bf32d58-38a3-4ae2-958e-57e9ca4a1e29 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-16 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3bf32d58-38a3-4ae2-958e-57e9ca4a1e29"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.37"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-76jz2 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] - name: kube-api-access-76jz2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to infer\ [e2e-llm-inference-service] \ RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/* 2>/dev/null\n\ [e2e-llm-inference-service] \ grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/* 2>/dev/null\n\ [e2e-llm-inference-service] \n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"$hca_dir\"\ [e2e-llm-inference-service] \ ]; then\n hca_name=$(basename \"$hca_dir\")\n port_state_file=\"\ [e2e-llm-inference-service] $hca_dir/ports/1/state\" # Assume port 1\n type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\ [e2e-llm-inference-service] \n\n echo \"[Infer RoCE] Check if the port state file ${port_state_file}\ [e2e-llm-inference-service] \ exists and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] &&\ [e2e-llm-inference-service] \ grep -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found active\ [e2e-llm-inference-service] \ HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n else\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Skipping inactive or down HCA: $hca_name\"\ [e2e-llm-inference-service] \n fi\n fi\n done\n\n # Check if we found any active HCAs\n\ [e2e-llm-inference-service] \ if [ ${#active_hcas[@]} -gt 0 ]; then\n # Join the array elements with\ [e2e-llm-inference-service] \ a comma\n hca_port_pairs=()\n for hca in \"${active_hcas[@]}\";\ [e2e-llm-inference-service] \ do\n hca_port_pairs+=(\"${hca}:1\")\n done\n\n active_hca_list=$(IFS=,;\ [e2e-llm-inference-service] \ echo \"${active_hcas[*]}\")\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\"\ [e2e-llm-inference-service] )\n echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n\ [e2e-llm-inference-service] \ export UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found. NCCL_IB_HCA\ [e2e-llm-inference-service] \ will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Finding GID_INDEX for each active HCA (SR-IOV compatible)...\"\ [e2e-llm-inference-service] \n\n # For SR-IOV environments, find the most common IPv4 RoCE v2 GID index\ [e2e-llm-inference-service] \ across all HCAs\n declare -A gid_index_count\n declare -A hca_gid_index\n\ [e2e-llm-inference-service] \n for hca_name in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Processing HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for\ [e2e-llm-inference-service] \ this HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"$tpath\"\ [e2e-llm-inference-service] \ 2>/dev/null; then\n idx=$(basename \"$tpath\")\n \ [e2e-llm-inference-service] \ gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n \ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo \"\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Found IPv4 RoCE v2 GID for ${hca_name}:\ [e2e-llm-inference-service] \ index=${idx}, gid=${gid_value}\"\n hca_gid_index[\"${hca_name}\"\ [e2e-llm-inference-service] ]=\"${idx}\"\n gid_index_count[\"${idx}\"]=$((${gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]} + 1))\n break # Use first found IPv4 GID per\ [e2e-llm-inference-service] \ HCA\n fi\n fi\n done\n done\n\n\ [e2e-llm-inference-service] \ # Find the most common GID index (most likely to be consistent across\ [e2e-llm-inference-service] \ nodes)\n best_gid_index=\"\"\n max_count=0\n for idx in \"\ [e2e-llm-inference-service] ${!gid_index_count[@]}\"; do\n count=${gid_index_count[\"${idx}\"]}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n \ [e2e-llm-inference-service] \ if [ $count -gt $max_count ]; then\n max_count=$count\n\ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n #\ [e2e-llm-inference-service] \ Use deterministic fallback if tied - prefer index 3 (SR-IOV standard)\n \ [e2e-llm-inference-service] \ if [ ${#gid_index_count[@]} -gt 1 ]; then\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ Multiple GID indices found, selecting most common: ${best_gid_index}\"\n \ [e2e-llm-inference-service] \ # If there's a tie, prefer index 3 as it's most common in SR-IOV setups\n\ [e2e-llm-inference-service] \ if [ -n \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\"\ [e2e-llm-inference-service] \ -eq \"$max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for NCCL,\ [e2e-llm-inference-service] \ NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR: No valid\ [e2e-llm-inference-service] \ IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any HCA.\"\n \ [e2e-llm-inference-service] \ fi\n else\n echo \"[Infer RoCE] No active HCAs found, skipping GID_INDEX\ [e2e-llm-inference-service] \ inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints landed in vLLM\ [e2e-llm-inference-service] \ 0.16.0 (vllm-project/vllm#30011).\n# Older versions still need the blanket\ [e2e-llm-inference-service] \ --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+ ]] &&\ [e2e-llm-inference-service] \ [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort -V | head\ [e2e-llm-inference-service] \ -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout 40\"\ [e2e-llm-inference-service] \nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name \"facebook/opt-125m\"\ [e2e-llm-inference-service] \ \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\" \\\n --port 8000\ [e2e-llm-inference-service] \ \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS} \\\n --enable-ssl-refresh\ [e2e-llm-inference-service] \ \\\n --ssl-certfile /var/run/kserve/tls/tls.crt \\\n --ssl-keyfile /var/run/kserve/tls/tls.key\ [e2e-llm-inference-service] \ \\\n ${VLLM_ADDITIONAL_ARGS} \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: kube-api-access-76jz2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fg56t [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:48Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] hostIP: 10.0.132.16 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.16 [e2e-llm-inference-service] podIP: 10.134.0.37 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.37 [e2e-llm-inference-service] startTime: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] finishedAt: '2026-07-09T18:03:48Z' [e2e-llm-inference-service] containerID: cri-o://96e21b2772a740494c6a78d3731965dd442800d285dfa2743489fec1f489d206 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] containerID: cri-o://96e21b2772a740494c6a78d3731965dd442800d285dfa2743489fec1f489d206 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-76jz2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T18:03:48Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] imageID: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo@sha256:afb39fca138b51d019d986229d546531b45a2a3deb73bcf59bd42406e13fbba0 [e2e-llm-inference-service] containerID: cri-o://3ede921af921e7f35aa4d5153110238108ca9cba627a25c66cfdb726893bae27 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-76jz2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-684fcd8f47- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6c5e7685-a76b-42fa-8048-fb682f709bfe [e2e-llm-inference-service] resourceVersion: '33374' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 684fcd8f47 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.41/23"],"mac_address":"0a:58:0a:85:00:29","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.41/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.41\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:29\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-684fcd8f47 [e2e-llm-inference-service] uid: 8abd23ae-3534-4c39-90c2-0e698a59804d [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-141-44 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"8abd23ae-3534-4c39-90c2-0e698a59804d"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.41"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-q75mr [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-q75mr [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: max-score-picker\n- type: lora-affinity-scorer\nschedulingProfiles:\n\ [e2e-llm-inference-service] - name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n weight:\ [e2e-llm-inference-service] \ 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-q75mr [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fg56t [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nnmc9 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:46Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] hostIP: 10.0.141.44 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.141.44 [e2e-llm-inference-service] podIP: 10.133.0.41 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.41 [e2e-llm-inference-service] startTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] state: [e2e-llm-inference-service] terminated: [e2e-llm-inference-service] exitCode: 0 [e2e-llm-inference-service] reason: Completed [e2e-llm-inference-service] startedAt: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] finishedAt: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] containerID: cri-o://7c5534907c6b8b36b7dab582ccd1c0f45aef51f6ce9eec0661418a78e9e27949 [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] containerID: cri-o://7c5534907c6b8b36b7dab582ccd1c0f45aef51f6ce9eec0661418a78e9e27949 [e2e-llm-inference-service] started: false [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] - name: kube-api-access-q75mr [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T18:03:46Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://2b93fe0fb95f6ba7f790f7a8a5dc30e40027c5fc604b3e284f5856f20b916b0a [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-q75mr [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: c9634621-1814-4926-b8ac-198d3f440fbe [e2e-llm-inference-service] resourceVersion: '32621' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nnmc9 [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nnmc9"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-fg56t"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-fg56t [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nnmc9 [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-fg56t [e2e-llm-inference-service] - name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nnmc9 [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b5f5d4fd-18b5-40a2-8aa3-9a9b893eb3b1 [e2e-llm-inference-service] resourceVersion: '32642' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.32.93 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.32.93 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 87070190-b7c2-4e07-a989-e1ade156e9c6 [e2e-llm-inference-service] resourceVersion: '32615' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.228.154 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.228.154 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: bfeaca77-4c84-4788-b482-18cd86ee3155 [e2e-llm-inference-service] resourceVersion: '34162' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6ca145dc-6364-4211-83e1-66ddbce7bb21 [e2e-llm-inference-service] resourceVersion: '33379' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\n- type: lora-affinity-scorer\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n\ [e2e-llm-inference-service] \ weight: 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-684fcd8f47" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3bf32d58-38a3-4ae2-958e-57e9ca4a1e29 [e2e-llm-inference-service] resourceVersion: '34161' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 78595fb58d [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] uid: bfeaca77-4c84-4788-b482-18cd86ee3155 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"bfeaca77-4c84-4788-b482-18cd86ee3155"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"TORCHINDUCTOR_CACHE_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"USER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_CPU_KVCACHE_SPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_ENABLE_V1_MULTIPROCESSING"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_CONFIGMAP_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"CA_BUNDLE_VOLUME_MOUNT_POINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/etc/ssl/custom-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"cabundle-cert"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:configMap: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 78595fb58d [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 78595fb58d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] configMap: [e2e-llm-inference-service] name: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] - hf://edbeeching/opt-125m-lora [e2e-llm-inference-service] - /mnt/lora/lora-adapter-1 [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: CA_BUNDLE_CONFIGMAP_NAME [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: CA_BUNDLE_VOLUME_MOUNT_POINT [e2e-llm-inference-service] value: /etc/ssl/custom-certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] - name: cabundle-cert [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /etc/ssl/custom-certs [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/bash [e2e-llm-inference-service] - -c [e2e-llm-inference-service] - "if [ -f /etc/profile.d/ibm-aiu-setup.sh ]; then\n source /etc/profile.d/ibm-aiu-setup.sh\n\ [e2e-llm-inference-service] fi\n\nif [ \"$KSERVE_INFER_ROCE\" = \"true\" ]; then\n echo \"Trying to\ [e2e-llm-inference-service] \ infer RoCE configs ... \"\n grep -H . /sys/class/infiniband/*/ports/*/gids/*\ [e2e-llm-inference-service] \ 2>/dev/null\n grep -H . /sys/class/infiniband/*/ports/*/gid_attrs/types/*\ [e2e-llm-inference-service] \ 2>/dev/null\n\n cat /proc/driver/nvidia/params\n\n KSERVE_INFER_IB_GID_INDEX_GREP=${KSERVE_INFER_IB_GID_INDEX_GREP:-\"\ [e2e-llm-inference-service] RoCE v2\"}\n\n echo \"[Infer RoCE] Discovering active HCAs ...\"\n active_hcas=()\n\ [e2e-llm-inference-service] \ # Loop through all mlx5 devices found in sysfs\n for hca_dir in /sys/class/infiniband/mlx5_*;\ [e2e-llm-inference-service] \ do\n # Ensure it's a directory before proceeding\n if [ -d \"\ [e2e-llm-inference-service] $hca_dir\" ]; then\n hca_name=$(basename \"$hca_dir\")\n \ [e2e-llm-inference-service] \ port_state_file=\"$hca_dir/ports/1/state\" # Assume port 1\n \ [e2e-llm-inference-service] \ type_file=\"$hca_dir/ports/1/gid_attrs/types/*\"\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] Check if the port state file ${port_state_file} exists\ [e2e-llm-inference-service] \ and contains 'ACTIVE'\"\n if [ -f \"$port_state_file\" ] && grep\ [e2e-llm-inference-service] \ -q \"ACTIVE\" \"$port_state_file\" && grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\"\ [e2e-llm-inference-service] \ ${type_file} 2>/dev/null; then\n echo \"[Infer RoCE] Found\ [e2e-llm-inference-service] \ active HCA: $hca_name\"\n active_hcas+=(\"$hca_name\")\n\ [e2e-llm-inference-service] \ else\n echo \"[Infer RoCE] Skipping inactive or\ [e2e-llm-inference-service] \ down HCA: $hca_name\"\n fi\n fi\n done\n\n # Check if\ [e2e-llm-inference-service] \ we found any active HCAs\n if [ ${#active_hcas[@]} -gt 0 ]; then\n \ [e2e-llm-inference-service] \ # Join the array elements with a comma\n hca_port_pairs=()\n \ [e2e-llm-inference-service] \ for hca in \"${active_hcas[@]}\"; do\n hca_port_pairs+=(\"\ [e2e-llm-inference-service] ${hca}:1\")\n done\n\n active_hca_list=$(IFS=,; echo \"${active_hcas[*]}\"\ [e2e-llm-inference-service] )\n hca_port_pairs_list=$(IFS=,; echo \"${hca_port_pairs[*]}\")\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Setting active HCAs: ${active_hca_list}\"\n \ [e2e-llm-inference-service] \ export NCCL_IB_HCA=${NCCL_IB_HCA:-${active_hca_list}}\n export\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST:-${hca_port_pairs_list}}\n export\ [e2e-llm-inference-service] \ UCX_NET_DEVICES=${UCX_NET_DEVICES:-${hca_port_pairs_list}}\n\n echo\ [e2e-llm-inference-service] \ \"[Infer RoCE] NCCL_IB_HCA=${NCCL_IB_HCA}\"\n echo \"[Infer RoCE]\ [e2e-llm-inference-service] \ NVSHMEM_HCA_LIST=${NVSHMEM_HCA_LIST}\"\n echo \"[Infer RoCE] UCX_NET_DEVICES=${UCX_NET_DEVICES}\"\ [e2e-llm-inference-service] \n else\n echo \"[Infer RoCE] WARNING: No active RoCE HCAs found.\ [e2e-llm-inference-service] \ NCCL_IB_HCA will not be set.\"\n fi\n\n if [ ${#active_hcas[@]} -gt\ [e2e-llm-inference-service] \ 0 ]; then\n echo \"[Infer RoCE] Finding GID_INDEX for each active\ [e2e-llm-inference-service] \ HCA (SR-IOV compatible)...\"\n\n # For SR-IOV environments, find\ [e2e-llm-inference-service] \ the most common IPv4 RoCE v2 GID index across all HCAs\n declare\ [e2e-llm-inference-service] \ -A gid_index_count\n declare -A hca_gid_index\n\n for hca_name\ [e2e-llm-inference-service] \ in \"${active_hcas[@]}\"; do\n echo \"[Infer RoCE] Processing\ [e2e-llm-inference-service] \ HCA: ${hca_name}\"\n\n # Find all RoCE v2 IPv4 GIDs for this\ [e2e-llm-inference-service] \ HCA and count by index\n for tpath in /sys/class/infiniband/${hca_name}/ports/1/gid_attrs/types/*;\ [e2e-llm-inference-service] \ do\n if grep -q \"${KSERVE_INFER_IB_GID_INDEX_GREP}\" \"\ [e2e-llm-inference-service] $tpath\" 2>/dev/null; then\n idx=$(basename \"$tpath\"\ [e2e-llm-inference-service] )\n gid_file=\"/sys/class/infiniband/${hca_name}/ports/1/gids/${idx}\"\ [e2e-llm-inference-service] \n # Check for IPv4 GID (contains ffff:)\n \ [e2e-llm-inference-service] \ if [ -f \"$gid_file\" ] && grep -q \"ffff:\" \"$gid_file\"; then\n\ [e2e-llm-inference-service] \ gid_value=$(cat \"$gid_file\" 2>/dev/null || echo\ [e2e-llm-inference-service] \ \"\")\n echo \"[Infer RoCE] Found IPv4 RoCE v2 GID\ [e2e-llm-inference-service] \ for ${hca_name}: index=${idx}, gid=${gid_value}\"\n \ [e2e-llm-inference-service] \ hca_gid_index[\"${hca_name}\"]=\"${idx}\"\n gid_index_count[\"\ [e2e-llm-inference-service] ${idx}\"]=$((${gid_index_count[\"${idx}\"]} + 1))\n \ [e2e-llm-inference-service] \ break # Use first found IPv4 GID per HCA\n fi\n \ [e2e-llm-inference-service] \ fi\n done\n done\n\n # Find the most common\ [e2e-llm-inference-service] \ GID index (most likely to be consistent across nodes)\n best_gid_index=\"\ [e2e-llm-inference-service] \"\n max_count=0\n for idx in \"${!gid_index_count[@]}\"; do\n\ [e2e-llm-inference-service] \ count=${gid_index_count[\"${idx}\"]}\n echo \"[Infer\ [e2e-llm-inference-service] \ RoCE] GID_INDEX ${idx} found on ${count} HCAs\"\n if [ $count\ [e2e-llm-inference-service] \ -gt $max_count ]; then\n max_count=$count\n \ [e2e-llm-inference-service] \ best_gid_index=\"$idx\"\n fi\n done\n\n # Use deterministic\ [e2e-llm-inference-service] \ fallback if tied - prefer index 3 (SR-IOV standard)\n if [ ${#gid_index_count[@]}\ [e2e-llm-inference-service] \ -gt 1 ]; then\n echo \"[Infer RoCE] Multiple GID indices found,\ [e2e-llm-inference-service] \ selecting most common: ${best_gid_index}\"\n # If there's a tie,\ [e2e-llm-inference-service] \ prefer index 3 as it's most common in SR-IOV setups\n if [ -n\ [e2e-llm-inference-service] \ \"${gid_index_count['3']}\" ] && [ \"${gid_index_count['3']}\" -eq \"\ [e2e-llm-inference-service] $max_count\" ]; then\n best_gid_index=\"3\"\n \ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using deterministic fallback: GID_INDEX=3 (SR-IOV\ [e2e-llm-inference-service] \ standard)\"\n fi\n fi\n\n # Check if GID_INDEX is already\ [e2e-llm-inference-service] \ set via environment variables\n if [ -n \"${NCCL_IB_GID_INDEX}\"\ [e2e-llm-inference-service] \ ]; then\n echo \"[Infer RoCE] Using pre-configured NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ from environment\"\n export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$NCCL_IB_GID_INDEX}\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Using pre-configured GID_INDEX=${NCCL_IB_GID_INDEX}\ [e2e-llm-inference-service] \ for NCCL, NVSHMEM, and UCX\"\n elif [ -n \"$best_gid_index\" ]; then\n\ [e2e-llm-inference-service] \ echo \"[Infer RoCE] Selected GID_INDEX: ${best_gid_index} (found\ [e2e-llm-inference-service] \ on ${max_count} HCAs)\"\n\n export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export NVSHMEM_IB_GID_INDEX=${NVSHMEM_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \ export UCX_IB_GID_INDEX=${UCX_IB_GID_INDEX:-$best_gid_index}\n\ [e2e-llm-inference-service] \n echo \"[Infer RoCE] Exported GID_INDEX=${best_gid_index} for\ [e2e-llm-inference-service] \ NCCL, NVSHMEM, and UCX\"\n else\n echo \"[Infer RoCE] ERROR:\ [e2e-llm-inference-service] \ No valid IPv4 ${KSERVE_INFER_IB_GID_INDEX_GREP} GID_INDEX found on any\ [e2e-llm-inference-service] \ HCA.\"\n fi\n else\n echo \"[Infer RoCE] No active HCAs found,\ [e2e-llm-inference-service] \ skipping GID_INDEX inference.\"\n fi\nfi\n\n# --disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ landed in vLLM 0.16.0 (vllm-project/vllm#30011).\n# Older versions still\ [e2e-llm-inference-service] \ need the blanket --disable-uvicorn-access-log.\nACCESS_LOG_ARGS=\"--disable-uvicorn-access-log\"\ [e2e-llm-inference-service] \nVLLM_VERSION=$(vllm --version 2>/dev/null | tail -1 | awk '{print $NF}')\n\ [e2e-llm-inference-service] echo \"[access-log-detect] vllm version='${VLLM_VERSION}'\"\nif [[ \"$VLLM_VERSION\"\ [e2e-llm-inference-service] \ =~ ^[0-9]+\\.[0-9]+ ]] && [ \"$(printf '%s\\n%s\\n' \"0.16.0\" \"${VLLM_VERSION}\"\ [e2e-llm-inference-service] \ | sort -V | head -1)\" = \"0.16.0\" ]; then\n ACCESS_LOG_ARGS=\"--disable-access-log-for-endpoints\ [e2e-llm-inference-service] \ /health,/metrics,/ping\"\nfi\necho \"[access-log-detect] selected ACCESS_LOG_ARGS='${ACCESS_LOG_ARGS}'\"\ [e2e-llm-inference-service] \n\n# --shutdown-timeout landed in vLLM 0.18.0 (vllm-project/vllm#36666).\n\ [e2e-llm-inference-service] SHUTDOWN_TIMEOUT_ARGS=\"\"\nif [[ \"$VLLM_VERSION\" =~ ^[0-9]+\\.[0-9]+\ [e2e-llm-inference-service] \ ]] && [ \"$(printf '%s\\n%s\\n' \"0.18.0\" \"${VLLM_VERSION}\" | sort\ [e2e-llm-inference-service] \ -V | head -1)\" = \"0.18.0\" ]; then\n SHUTDOWN_TIMEOUT_ARGS=\"--shutdown-timeout\ [e2e-llm-inference-service] \ 40\"\nfi\n\neval \"exec vllm serve /mnt/models \\\n --served-model-name\ [e2e-llm-inference-service] \ \"facebook/opt-125m\" \"publishers/kserve-ci-e2e-test/models/facebook/opt-125m\"\ [e2e-llm-inference-service] \ \\\n --port 8000 \\\n ${ACCESS_LOG_ARGS} \\\n ${SHUTDOWN_TIMEOUT_ARGS}\ [e2e-llm-inference-service] \ \\\n --enable-ssl-refresh \\\n --ssl-certfile /var/run/kserve/tls/tls.crt\ [e2e-llm-inference-service] \ \\\n --ssl-keyfile /var/run/kserve/tls/tls.key \\\n ${VLLM_ADDITIONAL_ARGS}\ [e2e-llm-inference-service] \ \\\n $@\"" [e2e-llm-inference-service] - -- [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --enable-lora [e2e-llm-inference-service] - --lora-modules [e2e-llm-inference-service] - '''{"name":"lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] - '''{"name":"publishers/kserve-ci-e2e-test/models/lora-adapter-1","path":"/mnt/lora/lora-adapter-1"}''' [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: DEBUG [e2e-llm-inference-service] - name: VLLM_CPU_KVCACHE_SPACE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: VLLM_ENABLE_V1_MULTIPROCESSING [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: USER [e2e-llm-inference-service] value: nonroot [e2e-llm-inference-service] - name: TORCHINDUCTOR_CACHE_DIR [e2e-llm-inference-service] value: /tmp/torchinductor-cache [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '2' [e2e-llm-inference-service] memory: 7Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /mnt [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-684fcd8f47 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8abd23ae-3534-4c39-90c2-0e698a59804d [e2e-llm-inference-service] resourceVersion: '33377' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 684fcd8f47 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] uid: 6ca145dc-6364-4211-83e1-66ddbce7bb21 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"6ca145dc-6364-4211-83e1-66ddbce7bb21"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"storage-initializer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"AWS_ACCESS_KEY_ID"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_CA_BUNDLE_CONFIGMAP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_ENDPOINT_URL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"AWS_SECRET_ACCESS_KEY"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:secretKeyRef: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_ENABLE_HF_TRANSFER"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_HIGH_PERFORMANCE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HF_XET_NUM_CONCURRENT_RANGE_GETS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_ENDPOINT"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_USE_HTTPS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"S3_VERIFY_SSL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"STORAGE_ALLOW_PATTERNS"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/mnt/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"kserve-provision-location"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 684fcd8f47 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 684fcd8f47 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: storage-initializer [e2e-llm-inference-service] image: quay.io/opendatahub/kserve-storage-initializer@sha256:6455858d9bcb13c4793d39d844d39ea45d90725b82962b5a6791b22458b6ca42 [e2e-llm-inference-service] args: [e2e-llm-inference-service] - hf://facebook/opt-125m [e2e-llm-inference-service] - /mnt/models [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_ACCESS_KEY_ID [e2e-llm-inference-service] - name: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] secretKeyRef: [e2e-llm-inference-service] name: seaweedfs-s3-creds [e2e-llm-inference-service] key: AWS_SECRET_ACCESS_KEY [e2e-llm-inference-service] - name: S3_USE_HTTPS [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: S3_ENDPOINT [e2e-llm-inference-service] value: s3-service.kserve:8333 [e2e-llm-inference-service] - name: AWS_ENDPOINT_URL [e2e-llm-inference-service] value: http://s3-service.kserve:8333 [e2e-llm-inference-service] - name: S3_VERIFY_SSL [e2e-llm-inference-service] value: '0' [e2e-llm-inference-service] - name: AWS_CA_BUNDLE [e2e-llm-inference-service] value: /etc/ssl/custom-certs/cabundle.crt [e2e-llm-inference-service] - name: AWS_CA_BUNDLE_CONFIGMAP [e2e-llm-inference-service] value: odh-kserve-custom-ca-bundle [e2e-llm-inference-service] - name: HF_HUB_ENABLE_HF_TRANSFER [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_HIGH_PERFORMANCE [e2e-llm-inference-service] value: '1' [e2e-llm-inference-service] - name: HF_XET_NUM_CONCURRENT_RANGE_GETS [e2e-llm-inference-service] value: '8' [e2e-llm-inference-service] - name: STORAGE_ALLOW_PATTERNS [e2e-llm-inference-service] value: '["tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", [e2e-llm-inference-service] "vocab.json", "merges.txt", "config.json", "generation_config.json"]' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 24Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 100m [e2e-llm-inference-service] memory: 100Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: kserve-provision-location [e2e-llm-inference-service] mountPath: /mnt/models [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: single-profile-handler\n- type: queue-scorer\n- type:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n- type: max-score-picker\n- type: lora-affinity-scorer\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: default\n plugins:\n - pluginRef: lora-affinity-scorer\n\ [e2e-llm-inference-service] \ weight: 4\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef:\ [e2e-llm-inference-service] \ prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] serviceAccount: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6a6d8aee-bfa6-433e-a5e5-f914c9bc6881 [e2e-llm-inference-service] resourceVersion: '32632' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7c94d43f-91ee-470f-a097-2c2f81685d5f [e2e-llm-inference-service] resourceVersion: '32627' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service-r566c [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 56087170-ce8a-4ba0-bb3c-5dfeb987cfce [e2e-llm-inference-service] resourceVersion: '33376' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] uid: b5f5d4fd-18b5-40a2-8aa3-9a9b893eb3b1 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:04:18Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b5f5d4fd-18b5-40a2-8aa3-9a9b893eb3b1"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.41 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr [e2e-llm-inference-service] uid: 6c5e7685-a76b-42fa-8048-fb682f709bfe [e2e-llm-inference-service] nodeName: ip-10-0-141-44.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svsth8k [e2e-llm-inference-service] generateName: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0b22563b-5f66-48ec-8d65-833020bbdbf2 [e2e-llm-inference-service] resourceVersion: '34159' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] uid: 87070190-b7c2-4e07-a989-e1ade156e9c6 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:05:04Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"87070190-b7c2-4e07-a989-e1ade156e9c6"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.37 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv [e2e-llm-inference-service] uid: ce9d7d7f-5c2b-4845-a794-a578803d5d9a [e2e-llm-inference-service] nodeName: ip-10-0-132-16.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 6a6d8aee-bfa6-433e-a5e5-f914c9bc6881 [e2e-llm-inference-service] resourceVersion: '32632' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7c94d43f-91ee-470f-a097-2c2f81685d5f [e2e-llm-inference-service] resourceVersion: '32627' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T18:03:43Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:04:06Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '33174' [e2e-llm-inference-service] uid: f2d77480-b883-4ba6-965b-f8ecca859668 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:04:06Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '33174' [e2e-llm-inference-service] uid: f2d77480-b883-4ba6-965b-f8ecca859668 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/lora-adapter-1 [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '33152' [e2e-llm-inference-service] uid: e3434eb4-a77d-4f1b-87a1-00810d82d39d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T18:03:46Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32806' [e2e-llm-inference-service] uid: cd5c8ce5-db09-49fa-9aec-1c127e146f3d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T18:03:46Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32697' [e2e-llm-inference-service] uid: d188d30b-6676-4be0-8ffe-3ce2b5da0235 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '33157' [e2e-llm-inference-service] uid: 50cae8df-f8d8-4326-bcc2-a34c6232d98b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32733' [e2e-llm-inference-service] uid: 61850610-d4c8-489e-8864-021502ca73bb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32697' [e2e-llm-inference-service] uid: d188d30b-6676-4be0-8ffe-3ce2b5da0235 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '33157' [e2e-llm-inference-service] uid: 50cae8df-f8d8-4326-bcc2-a34c6232d98b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32733' [e2e-llm-inference-service] uid: 61850610-d4c8-489e-8864-021502ca73bb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32697' [e2e-llm-inference-service] uid: d188d30b-6676-4be0-8ffe-3ce2b5da0235 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:04:05Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '33157' [e2e-llm-inference-service] uid: 50cae8df-f8d8-4326-bcc2-a34c6232d98b [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-p-ip-f5162b44.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:caCertificates: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:45Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32733' [e2e-llm-inference-service] uid: 61850610-d4c8-489e-8864-021502ca73bb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] caCertificates: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt [e2e-llm-inference-service] insecureSkipVerify: false [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e079e6c8-fff3-4615-a01b-145cec419e7f"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T18:03:44Z' [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] uid: e079e6c8-fff3-4615-a01b-145cec419e7f [e2e-llm-inference-service] resourceVersion: '32653' [e2e-llm-inference-service] uid: 019f1596-00f2-4e28-ba47-ff0d73fbd252 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:05:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: both [e2e-llm-inference-service] pod-template-hash: 78595fb58d [e2e-llm-inference-service] timestamp: '2026-07-09T18:05:02Z' [e2e-llm-inference-service] window: 16.473s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 1031785770n [e2e-llm-inference-service] memory: 2325952Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T18:05:19Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: llmisvc-model-fb-opt-125m-with-ba4d693a [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 684fcd8f47 [e2e-llm-inference-service] timestamp: '2026-07-09T18:04:55Z' [e2e-llm-inference-service] window: 16.161s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 517047n [e2e-llm-inference-service] memory: 21684Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-09T18:05:20.033858] end - ❌ 100.206s: Expected model 'lora-adapter-1' in data[].id, found: ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] assert 'lora-adapter-1' in ['facebook/opt-125m', 'publishers/kserve-ci-e2e-test/models/facebook/opt-125m'] [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] ====== 1 failed, 41 passed, 3 skipped, 29 warnings in 2844.46s (0:47:24) ======= [must-gather] [must-gather ] OUT 2026-07-09T18:40:01.397952102Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f73a085b-0234-456f-8f79-621ceb1f1251 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T18:40:01.447482847Z namespace/openshift-must-gather-2kvkt created [must-gather] [must-gather ] OUT 2026-07-09T18:40:01.452130556Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-7d2zk created [must-gather] [must-gather ] OUT 2026-07-09T18:40:01.472374587Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:08.828160346Z [disk usage checker] Started [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:08.831393415Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.102809868Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.262397823Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.262428565Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.262428565Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.265229267Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.442333138Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.442362959Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.442362959Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.444991089Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.613025892Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.613057237Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.613057237Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.615735887Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.778241954Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.778271721Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.778271721Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.781147047Z Error getting logs from rhods-notebooks [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.955558359Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.955648715Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.955648715Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:09.958309794Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.120110411Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.120139620Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.120139620Z namespaces "istio-system" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.124179855Z Error getting logs from istio-system [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.283819363Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.283865852Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.283865852Z namespaces "knative-serving" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.286655241Z Error getting logs from knative-serving [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.445369504Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.445461150Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.445461150Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.448055650Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:10.719537656Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.027585426Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.170312663Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.256621306Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.259114830Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.409871863Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.503797059Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.506577686Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.639539562Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.724181269Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.726774288Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.864235931Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.948162677Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:11.950659823Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.094549878Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.178987132Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.182685861Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.333032423Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.419223385Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.421915966Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.572406707Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.654099167Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.656512364Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.792486449Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.877949367Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:12.880780628Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.012304393Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.115333570Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.118690947Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.265569069Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.348512822Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.351170038Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.502363571Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.598958464Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.601567517Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.734677086Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.817632502Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.820456124Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.835989119Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:13.952548867Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.038280589Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.040830894Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.173365218Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.254504046Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.257059388Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.387016425Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.466310546Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.468639759Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.600125596Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.683696650Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.686301437Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.822692567Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.905332124Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:14.908025205Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:15.042540005Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:15.123807127Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:15.126150296Z Error collecting info from [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:15.860878859Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:16.008901232Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:16.135168363Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:16.270985671Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:16.408605010Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:17.030121677Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:17.175168995Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:17.313520857Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:17.456577958Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:17.581991402Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:18.135141087Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:18.520643683Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:18.840698370Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:19.270759989Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:19.685140422Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:19.846398414Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:20.211547242Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:20.952661024Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:21.350727822Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:21.517978087Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:21.835846417Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:22.601144234Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:22.959285342Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:23.125105653Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:23.492870365Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:23.845305412Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:24.312940776Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:24.679518323Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:24.856155872Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:25.165221746Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:25.937635608Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:26.300945321Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:26.476391801Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:26.837752971Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:27.620941194Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:28.006375262Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:28.181095190Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:28.545811818Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:28.850640296Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:29.352112621Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:29.751864518Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:29.913614398Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:30.234654878Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:31.021306656Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:31.434405834Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:31.597678813Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:31.929917153Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:32.705471792Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:33.089169355Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:33.263157629Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:33.586755575Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:33.855949417Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:34.322534967Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:34.704927229Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:34.899100567Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:35.223901793Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:35.976535095Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:36.361338247Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:36.561182092Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:36.874171576Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:37.618421048Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:38.000988902Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:38.161858179Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:38.484370232Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:38.861677504Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:39.213608619Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:39.563312699Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:39.724036909Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:40.053719342Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:40.823516018Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:41.193810407Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:41.371536986Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:41.671091069Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:42.437066362Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:42.813597040Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:42.976606327Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:43.266335256Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:43.870012364Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:44.012977620Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:44.389460104Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:44.561776502Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:45.689661623Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:45.856470995Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:46.269214075Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:46.431879812Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:47.563274660Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:47.742810328Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:48.158155596Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:48.329507751Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:48.875195557Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:49.461177767Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:49.613543322Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:50.021954266Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:50.184716130Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:51.327780378Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:51.527815516Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:51.908325201Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:52.083598328Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:52.717135755Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:52.939595974Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:53.350281898Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:53.519358530Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:53.881337189Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:54.142095209Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:54.357140240Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:54.817219789Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:54.980117742Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:55.609978964Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:55.786348747Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:56.195504091Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:56.356316809Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:56.975556566Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:57.167713338Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:57.576678937Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:57.744618283Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:58.364537635Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:58.555105559Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:58.887188971Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:58.965423969Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:59.144630497Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:59.777254420Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:40:59.966101638Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:00.389924135Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:00.549913852Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:01.160197214Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:01.355959993Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:01.797909664Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:01.959857725Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:02.563781918Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:02.776599364Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:03.204251801Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:03.381631476Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:03.892115230Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:04.015694515Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:04.212254498Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:04.632659682Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:04.797002261Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:05.400758587Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:05.594770212Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:06.024931963Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:06.190879731Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:06.818514045Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:07.006213180Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:07.418514321Z Wrote inspect data to must-gather. [must-gather] [must-gather-cg9r9] POD 2026-07-09T18:41:07.437107132Z Caches written to disk [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.449271447Z waiting for gather to complete [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.452766704Z downloading gather output [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.741569637Z receiving incremental file list [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.75893385Z ./ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.75897107Z aggregated-discovery-api.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.759249896Z aggregated-discovery-apis.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.760036121Z event-filter.html [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.762676051Z timestamp [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.762840235Z version [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763707091Z cluster-scoped-resources/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763722052Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763728952Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763772372Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763851884Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763874974Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763887525Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763949876Z namespaces/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763961106Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.763975726Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.764049398Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.764101949Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.76415103Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.76417296Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.764290432Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.764656009Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765048357Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765097108Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765157719Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.76520432Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765267821Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765378223Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765454545Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765534776Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765623148Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765666189Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.76570429Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765859402Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.765985985Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.771103583Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.771209975Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.771601412Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.771737775Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77197964Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.772088042Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.772143223Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773340346Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773424017Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773484588Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77356099Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773667592Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773730123Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773775644Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773827455Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773884406Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773930447Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773942767Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.773963728Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/e2e-pvc-model-download-q2skc.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774040779Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/storage-initializer/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774050019Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/storage-initializer/storage-initializer/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774055039Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.7740945Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774216173Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774286664Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-q2skc/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774321424Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.774369396Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77458773Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/main/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77460319Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/main/main/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77460767Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/main/main/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77461516Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/main/main/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776014427Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/main/main/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776097889Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/main/main/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776139569Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/storage-initializer/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.7761542Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/storage-initializer/storage-initializer/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77616082Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77619067Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776288752Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776353554Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-78595fb58d-pb8bv/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776372774Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776431325Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776591518Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/main/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776604788Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/main/main/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776610938Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/main/main/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776634379Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/main/main/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776797202Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/main/main/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776859773Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/main/main/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776894864Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/storage-initializer/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776910244Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/storage-initializer/storage-initializer/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776914934Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.776924674Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777033607Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777115958Z namespaces/kserve-ci-e2e-test/pods/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777155449Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777178889Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/router-gateway-1-openshift-default-54c47cdfc7-hb6xz.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777302312Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/istio-proxy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777311012Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/istio-proxy/istio-proxy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777315112Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777324752Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777458675Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-hb6xz/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777566557Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777621298Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/router-gateway-2-openshift-default-684dc9b4b9-nkv2c.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777719109Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/istio-proxy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77772722Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/istio-proxy/istio-proxy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77773135Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777766841Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777881273Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-nkv2c/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.777986195Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778026266Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778073906Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778113337Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778160658Z namespaces/kuadrant-system/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778206379Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77826769Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778306601Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778352842Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778393362Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778476654Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.778829151Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779118296Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779170177Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779201198Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779251469Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.77929638Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779373451Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779418892Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779460003Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779562725Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779609896Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779652037Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779816Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.779935962Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.780419671Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.780496843Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78088642Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781009563Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781237937Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781349919Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78139396Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781507442Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781562923Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781615444Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781646655Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781736797Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781756177Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781816758Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781868749Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78190001Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781954231Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781962791Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.781991562Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino-7c77957575-czzst.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.782071093Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.782086713Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.782091783Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.782119774Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785634651Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785704753Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785725173Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785787954Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/authorino-operator-6d85f6564-tsj82.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785876376Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785885416Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785890336Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.785901736Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786123771Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786189482Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786216503Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786279694Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/dns-operator-controller-manager-65b49595d7-rc99j.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786348725Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786358535Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786363435Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786371505Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786492538Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786580879Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78660017Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78663704Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin-85fd4b8b9d-rqdm2.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786756793Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786765813Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786769843Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786804734Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786876835Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786942206Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.786973007Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.787018978Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/kuadrant-operator-controller-manager-bb7ddb7d7-z679n.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78712849Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78713601Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.78713999Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.787173861Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805255367Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805327749Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805347819Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.80542576Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador-limitador-69574b596d-t6fxz.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805538093Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805553943Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805559493Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805599964Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805701786Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805786367Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805825278Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805887809Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/limitador-operator-controller-manager-6f9f468797-xh7g7.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805982231Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805989591Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.805993491Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806041132Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806220096Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806308287Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806353618Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806409839Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.80646703Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806539042Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806615113Z namespaces/openshift-ingress/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806678584Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806767516Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806809337Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806876928Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.806925819Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807017491Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807264386Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.80748585Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807559132Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807621603Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807686794Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807745995Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807837227Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807897698Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.807941619Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.808051551Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.808118472Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.808173373Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.808604801Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.808735284Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.808976859Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.80906932Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809322965Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809453508Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809681802Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809777854Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809817865Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809910306Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.809949297Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810019048Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810061819Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810165091Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810217242Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810275793Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810339355Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810393696Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810450317Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810458177Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810507598Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/istiod-openshift-gateway-94bb8fbfd-p9b99.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.81062947Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.81064034Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.81064472Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.810689831Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.873181529Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.87322483Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.87324019Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.873335641Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/openshift-ai-inference-openshift-default-866675cbb8-cvzqk.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.873478284Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.873505205Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.873514915Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.873584606Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876324589Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.87639151Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876414111Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router-default-5bc67f59d-8gfgw.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876561574Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876585424Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876594794Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876609074Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/current.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876731127Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/previous.insecure.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876769818Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/previous.log [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876798188Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.876853739Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.8768873Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.877031372Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.88522816Z [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.8852634Z sent 3,424 bytes received 2,250,820 bytes 4,508,488.00 bytes/sec [must-gather] [must-gather-cg9r9] OUT 2026-07-09T18:41:08.88527334Z total size is 27,287,736 speedup is 12.11 [must-gather] [must-gather ] OUT 2026-07-09T18:41:09.123902042Z namespace/openshift-must-gather-2kvkt deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f73a085b-0234-456f-8f79-621ceb1f1251 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T18:41:09.247765395Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f73a085b-0234-456f-8f79-621ceb1f1251 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T18:41:09.268642096Z namespace/openshift-must-gather-xlxbw created [must-gather] [must-gather ] OUT 2026-07-09T18:41:09.276008737Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-m722v created [must-gather] [must-gather ] OUT 2026-07-09T18:41:09.298874815Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec435553f79197f28a94d55da851b8302383238997d790ed3335149e3ee4341f created [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:10.822260211Z [disk usage checker] Started [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:10.825549865Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:11.054241936Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:11.224896900Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:11.339064252Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:11.437421255Z Gathering data for ns/default... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:11.803343622Z Gathering data for ns/openshift... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:12.286817460Z Gathering data for ns/kube-system... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.078254505Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.429734973Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.429766903Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.429766903Z namespaces "assisted-installer" not found [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.851948523Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.861675796Z Executing Istio gather script [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.862143066Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.867591248Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.876821111Z INFO: Collecting host service logs for crio [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.877740702Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.878239698Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.878664292Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.879154545Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.879549021Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.879942485Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.880280828Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.880613358Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.881020880Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.881353234Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.881722721Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.887535076Z INFO: Gathering HAProxy config files [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.894066047Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:13.965029467Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:14.299777595Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:14.911621235Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.029110132Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.453874734Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.453912075Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.581887521Z INFO: Collecting Insights Archives from insights-operator-68d9dd94cf-ft7pk [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.581940536Z insights-runtime-extractor-5vv52 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.582021631Z insights-runtime-extractor-fsgn4 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.582060730Z insights-runtime-extractor-zpv9t [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.739443858Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.783446172Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:15.897688781Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.100583852Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.186415074Z No resources found [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.222631488Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.294737289Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.302561674Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.381703455Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.469515160Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.790322314Z No resources found [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.897476287Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:16.927075579Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.030696516Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.030696516Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.062952680Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.253763582Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.302030558Z INFO: 'previous-logs' folder not found on ip-10-0-132-16.ec2.internal, skipping... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.369051625Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.428683743Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.438161599Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.546919271Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.551636767Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.552145861Z INFO: HAProxy config collection complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.710179696Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.847094008Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.924316620Z INFO: 'previous-logs' folder not found on ip-10-0-141-118.ec2.internal, skipping... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.977121451Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:17.977773683Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.039652906Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.329002197Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.365335502Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.563358198Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.583605370Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.610920183Z INFO: 'previous-logs' folder not found on ip-10-0-141-44.ec2.internal, skipping... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.611022340Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.611074656Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.799664329Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.874658826Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:18.879853683Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.094211338Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.189638384Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.207582396Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.355737064Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.367801250Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.430308684Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.524868765Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.706236187Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.720096768Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.785278933Z INFO: INTERCONNECT MODE [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.785545105Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.882868418Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.885607736Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:19.979519373Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.040942956Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.067264575Z INFO: Gathering OVN_Northbound from ovnkube-node-jwnj6... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.068399988Z INFO: Gathering OVN_Northbound from ovnkube-node-sc5lg... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.068399988Z INFO: Gathering OVN_Northbound from ovnkube-node-sh5ck... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.068399988Z INFO: Gathering OVN_Southbound from ovnkube-node-jwnj6... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.068399988Z INFO: Gathering OVN_Southbound from ovnkube-node-sc5lg... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.069171545Z INFO: Gathering OVN_Southbound from ovnkube-node-sh5ck... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.125407443Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.372680666Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.376905504Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.454457612Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.539877004Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.541997047Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.575686945Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.580745600Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.582440211Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.584694130Z tar: Removing leading `/' from member names [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.587562138Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.743222068Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.908862447Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.937053847Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:20.996210073Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.214929612Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.214929612Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.215631511Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.218071241Z 7.2M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.221408277Z ovnk_database_store/ [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.221408277Z ovnk_database_store/ovnkube-node-sh5ck_sbdb [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.269026965Z ovnk_database_store/ovnkube-node-sh5ck_nbdb [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.296954712Z ovnk_database_store/ovnkube-node-sc5lg_nbdb [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.317723266Z ovnk_database_store/ovnkube-node-sc5lg_sbdb [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.374294087Z ovnk_database_store/ovnkube-node-jwnj6_nbdb [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.392168839Z ovnk_database_store/ovnkube-node-jwnj6_sbdb [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.395266108Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.434966069Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.452112287Z INFO: Network log collection complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.718327191Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:21.779911326Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:22.037669851Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:22.283233276Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:22.559588682Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:22.562572471Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:22.788422615Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.121544503Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.146555117Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d8d5dc4ea635b13544e6f92103089219e732369ae30557b7beee2e8a11bf500d [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.327322611Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.377242936Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.381234483Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.501049708Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.641577105Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.651178029Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.987644534Z Gathering data for ns/openshift-console... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:23.992136045Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.240644222Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.243078725Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.412922285Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.675457861Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.678419436Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.690431368Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.763080042Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:24.894913682Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.206294518Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.209619522Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.396689532Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.435816127Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.690401639Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.692897015Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.841787546Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.862511857Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.928073083Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:25.943110219Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.148959304Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.154093745Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.357301362Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.592391956Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.595537786Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.597525981Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.855341718Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:26.995200867Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.084000559Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.087864064Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.199500561Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.310702611Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.541544307Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.543940334Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.692456251Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.724176930Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.959043137Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:27.962581649Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.136968069Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.178676107Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.413377806Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.416805072Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.597662695Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.852862592Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:28.856063928Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.044178278Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.262055382Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.303330653Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.307800209Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.314447197Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.503547849Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.761660044Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.765686305Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.803418194Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:29.947359691Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.159791506Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.163438702Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.372856473Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.470360677Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.633964225Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.638096740Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.832094320Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:30.949202095Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.033119280Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.035479834Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.219590593Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.418948501Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.421459466Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.624813764Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.626593958Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.865701187Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.868312922Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.974000230Z Collecting performance related data for node ip-10-0-132-16.ec2.internal [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.983001813Z Collecting performance related data for node ip-10-0-141-44.ec2.internal [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:31.990383457Z Collecting performance related data for node ip-10-0-141-118.ec2.internal [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:32.122981223Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:32.373487562Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:32.378380423Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:32.654528056Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:32.987453455Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:32.991293530Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:33.257693222Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:33.268472895Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:33.572646569Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:33.577140578Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:33.844374286Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:34.140559598Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:34.144297566Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:34.176313446Z Gathering data for ns/kserve... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:34.334872497Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:34.584495974Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:34.749693745Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:35.036121805Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:35.955194563Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:36.308127906Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:36.511935520Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:36.659272364Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:36.820904299Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:37.049993201Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:37.190168510Z Gathering data for ns/openshift-config... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:37.594556838Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:38.187830604Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:38.356469949Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:38.534973894Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:38.716987951Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:38.782130172Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:39.247138107Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:39.254898793Z Done executing Istio gather script [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:39.618390040Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:40.010813902Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:40.378814870Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:40.798173419Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:40.959824304Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:41.205148023Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:41.626368170Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:41.843559323Z Collecting kubelet logs for node ip-10-0-132-16.ec2.internal [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:41.846113623Z Collecting kubelet logs for node ip-10-0-141-44.ec2.internal [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:41.849150025Z Collecting kubelet logs for node ip-10-0-141-118.ec2.internal [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:42.352259648Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-xlxbw namespace [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:42.360029899Z INFO: Node performance data collection complete. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:42.454170992Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:43.906103569Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:45.295539982Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:45.670072669Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:45.966720467Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:46.274985375Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:46.786025866Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:47.302006875Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:47.673432604Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:48.205624139Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:48.621292149Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:49.162903446Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:49.499413071Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:49.877861567Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:50.602273212Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:50.971807504Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:50.981705276Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:51.338695438Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:51.739431392Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:52.732052432Z Wrote inspect data to must-gather. [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:52.732093739Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:52.732093739Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:52.885589228Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-5lnvf] POD 2026-07-09T18:41:53.072307492Z Caches written to disk [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.080965235Z waiting for gather to complete [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.084319699Z downloading gather output [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.372015318Z receiving incremental file list [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.39396633Z ./ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.394114683Z aggregated-discovery-api.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.394432639Z aggregated-discovery-apis.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.395472529Z event-filter.html [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.398231182Z timestamp [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.398325104Z version [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.404949282Z cluster-scoped-resources/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.404988312Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405037943Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405112025Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405235647Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405347269Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405428841Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405604174Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405720496Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405817218Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.405959071Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.406013112Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.406136544Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.406206046Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.406656964Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.406702815Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.406897389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.407153564Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.407357388Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.407498451Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.411237852Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.412884754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.413026077Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.41316848Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.413322083Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.413501056Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.414025626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.414184799Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.414364352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.414493795Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.41530525Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.415581556Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.415718749Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.415917202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.417103095Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.41734418Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.417533143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.417740637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.418052653Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.418201626Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.418425881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.418862549Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.419572263Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.419709715Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.419838728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.41997201Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.420100373Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.420224545Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.420403379Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.420629063Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.421439679Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.421607162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.421755545Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.423676762Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.423848895Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.424017148Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.424283503Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.424599109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.425039677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.42517579Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.425331493Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.425455376Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.425602368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.425747131Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.425867733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.426044297Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.42619543Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.426321592Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.426617188Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.426795521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.426924084Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.427478175Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.427637887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.42777132Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.427978664Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.428870751Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.429011824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.429262259Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.429394411Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.429598185Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.429734108Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.429904601Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430058284Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430192987Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430328709Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430474252Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430695396Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430773838Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.430970191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.431112544Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.431271427Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.431443401Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.431833778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.432025902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.43245239Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.432610333Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.433397708Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.433559361Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.433697274Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.433822966Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.433962239Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.434189644Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.434448309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.434613932Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.434746294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.434917487Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.43505901Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.435194453Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.435349206Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.435512309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.435672502Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.437958396Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.438431745Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.439025486Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.4392107Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.439395584Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.439570537Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.43973349Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.439861902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.439975865Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.440835441Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.441144547Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.444041913Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.444200066Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.447788765Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.447937018Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.448095721Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.448225093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.448359286Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.451018597Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.451310642Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.459448129Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.463722241Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.463890874Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.46419697Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.464388554Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.464569737Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.46469038Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.464872553Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.465153419Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.465275001Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.465440824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.465672199Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.465820852Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.465953634Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.466089027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.466220679Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.466390592Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.466553766Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.466716119Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.466843951Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.467005114Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.467171958Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.46729205Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.467446733Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.467600996Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.467744949Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.468027334Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.468185267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.468459012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.468594885Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.468727497Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.468836599Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.470817868Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.470965Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.471091143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.471213395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.471337518Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.471508821Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.471657654Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.471827767Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.47196693Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.473609561Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.473763324Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.473917237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.47406368Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.474198633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.474393906Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.474659481Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.475194402Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.475486037Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.47563828Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.475763983Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.475892325Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.476531137Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.476765062Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.476899824Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.478337042Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.478513685Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.478687079Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.478808651Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.478985514Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.47927845Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.479420663Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.479612557Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.480031394Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.480162517Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48029836Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.480504143Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.480690887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48084003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.480994173Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.481203527Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.481537843Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.481597095Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.481707897Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.481802679Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48190073Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.481993882Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.482161795Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.482277368Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48239321Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.482484772Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.482601534Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.482708046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.482811438Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48291211Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483016792Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483120474Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483219126Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483323538Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.4834251Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483534692Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483718565Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483816227Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.483922439Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484021841Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484136953Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484234375Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484339027Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484440419Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484562242Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484666484Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484767416Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.484865178Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48496698Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485072691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485175393Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485279065Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485385388Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485478919Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485644732Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485747494Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485848436Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.485950698Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48605728Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486160482Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486268274Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486369976Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486476109Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486602961Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486731323Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486860876Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.486947707Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.4870546Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487157231Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487265044Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487419407Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487485188Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48760811Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487709692Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487827424Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.487931377Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488035968Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488160591Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488265113Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488370305Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488472697Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48862479Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488731412Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488840774Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.488956356Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489060388Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.48917197Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489274392Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489372054Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489493107Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489614009Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489718491Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489830123Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.489931315Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490036447Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490150219Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490250881Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490355993Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490460675Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490583167Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490757761Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490866243Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.490970975Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491085907Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491201889Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491305911Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491417043Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491537886Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491666758Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.49177014Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491880082Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.491984424Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492109677Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492216479Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492318121Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492428183Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492532225Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492667617Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492759989Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492864821Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.492969403Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493085045Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493190238Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493243608Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493254899Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.49330649Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-dcsnf.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493455852Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-npt6v.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493614526Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-r4prr.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493809509Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.493881021Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.494042704Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.494194457Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.494298429Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.494438031Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.494790778Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.494918931Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495024643Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495145815Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495270458Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495367169Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495469001Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495659795Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495757247Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495857839Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.495971111Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496081973Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496196415Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496292567Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496396099Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496495731Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496620184Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496717625Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496832657Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496884489Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.496998931Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497106433Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497209975Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497311677Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497418909Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497530591Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497718465Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497821036Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.497927309Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498035461Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498140633Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498294225Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498397878Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.49851118Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498634872Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498737644Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498862627Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.498963038Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499074931Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499176842Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499263284Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499318915Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499420277Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499470158Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.49956286Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499604481Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499671912Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499723163Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499801834Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499847155Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499909007Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.499961078Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500033069Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500044869Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50010746Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500215862Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500314444Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500379246Z cluster-scoped-resources/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500389896Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500451537Z cluster-scoped-resources/core/nodes/ip-10-0-132-16.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500658741Z cluster-scoped-resources/core/nodes/ip-10-0-141-118.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500809804Z cluster-scoped-resources/core/nodes/ip-10-0-141-44.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500937786Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.500994108Z cluster-scoped-resources/core/persistentvolumes/pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501069619Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50114518Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501204781Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501310683Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501412285Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501510867Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501690071Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501790623Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.501960616Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502065408Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50218902Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502283142Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502391964Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502508337Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502632179Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502736971Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502838753Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.502940385Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503043957Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503150879Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503256961Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503352933Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503454825Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503578667Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503631098Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50375219Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503833712Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.503929084Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504029786Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504129818Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504300041Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504444624Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504569666Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504647548Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504659098Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504707009Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504794081Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504804751Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504845331Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504940973Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.504990944Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505051126Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505067216Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505137017Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505229709Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505397842Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505460013Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505470614Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505539885Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505601976Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505613506Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505662277Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505751249Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50579098Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505913262Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.505963713Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506051655Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506103516Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506182407Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506239268Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506549354Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506579125Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506588095Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506591915Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506629476Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506676977Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506745298Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506800309Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50687885Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.506938582Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507018533Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507074234Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507134586Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507179116Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507240357Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507290049Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50735242Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50740114Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507499973Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507510173Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507570284Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507666526Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507720417Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507853679Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.507962381Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508153655Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508291358Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508511682Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508652765Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508752607Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508825778Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508888209Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.508976581Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509081393Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509233276Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509376759Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509479631Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509677544Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509768046Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.509854728Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.50994298Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510028271Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510118483Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510200134Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510280946Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510365088Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510450689Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510556971Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510664183Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510744135Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510831126Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.510914488Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511034061Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511049681Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511163903Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511228334Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511317806Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511480539Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511592091Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511669323Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511757494Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511840396Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.511923618Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512006909Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512111061Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512192043Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512278074Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512364066Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512454838Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51256675Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512654541Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512740373Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512832835Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.512911906Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513004958Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51309055Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513174242Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513261713Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513355285Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513436557Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513536529Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51362521Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513722772Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513772353Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513782723Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513812634Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513874315Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513885465Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513898796Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513975167Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.513994077Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514030798Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514097419Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51413893Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-132-16.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514228842Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-141-118.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514305633Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-141-44.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514373595Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514422816Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514495677Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514583389Z host_service_logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514600249Z host_service_logs/masters/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51464172Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514707391Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514768692Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514834483Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.514958776Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515062058Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515126729Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51519777Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515262412Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515333183Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515397794Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515448235Z ingress_controllers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515459306Z ingress_controllers/default/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515470236Z ingress_controllers/default/router-default-5bc67f59d-8gfgw/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515504886Z ingress_controllers/default/router-default-5bc67f59d-8gfgw/haproxy.config [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515662949Z insights-data/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51567896Z istio/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515768021Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.515819313Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516305602Z istio/event-filter.html [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516446154Z istio/timestamp [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516595427Z istio/cluster-scoped-resources/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516607618Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516623218Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516664409Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51674103Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516780881Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516840032Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516924604Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.516966635Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.517132838Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.517977454Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.518172388Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.518402552Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.51881277Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.518985833Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.519371971Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520194757Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520313319Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520448441Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520604614Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520742907Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520861349Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.520985212Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.521085164Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.521288087Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.521351089Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.521541443Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.521808237Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.522002331Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.52245376Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.522613553Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.522739185Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.522877708Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.522887288Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.522934649Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523008211Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523018801Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523056341Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523158623Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523306496Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523377438Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523426879Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.52351412Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523699694Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523771785Z istio/namespaces/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523788756Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523796066Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523801016Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523841377Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.523958129Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524054931Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524114252Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524235154Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524326596Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524412228Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524497929Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524623022Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524634512Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524669952Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524731154Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524740724Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524769865Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/llmisvc-model-fb-opt-125m-with-ba4d693a-inference-pool.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524826755Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524835666Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524888687Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-scheduler.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.524992789Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525143902Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525228013Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525288135Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525412327Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525472688Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525601001Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525655271Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525741593Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.525966277Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526196502Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526243793Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526286574Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526356465Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526399576Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526481587Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526556199Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.52661371Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526770853Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526836174Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.526884455Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.527439986Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.52767004Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.527935435Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.528037977Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.528321723Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.528462896Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.528745671Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.528962295Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529044867Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529132928Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529142038Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529188839Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529296181Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529343893Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529451244Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529475985Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529605898Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529686049Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.52972632Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529779011Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529825862Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529835882Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.529904713Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530012625Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530151938Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.5302777Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530311471Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530396933Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530404573Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530456994Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/istiod-openshift-gateway-94bb8fbfd-p9b99.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530618687Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530629757Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530640057Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.530689508Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565406286Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565467237Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565526478Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565603459Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/openshift-ai-inference-openshift-default-866675cbb8-cvzqk.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565769782Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565779043Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565793823Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.565927816Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568405803Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568495295Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568552256Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router-default-5bc67f59d-8gfgw.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568679249Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568686919Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568693359Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.56874865Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568884392Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568940414Z istio/namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.568955174Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569000865Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569039375Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569091056Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569142307Z machine_config_ondisk/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569152998Z machine_config_termination_logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569161578Z monitoring/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569166328Z monitoring/alertmanager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569236809Z monitoring/alertmanager/status.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569345241Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569432593Z monitoring/prometheus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569471774Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569593776Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.569656617Z monitoring/prometheus/rules.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57029578Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57033078Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.570377401Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571240348Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571309509Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.5713445Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571532383Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571616735Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571716537Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571731417Z monitoring/prometheus/status/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.571780918Z monitoring/prometheus/status/config.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572240477Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572286948Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57238799Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572598494Z namespaces/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572610434Z namespaces/cert-manager-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572614554Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572618414Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572645705Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572717636Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572725066Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572773857Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.572988701Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.574699504Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.574851227Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575075831Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575291965Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575338337Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-v4bt8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575490979Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57553842Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575614642Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575671683Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575732664Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575759585Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575845886Z namespaces/cert-manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575859416Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575873157Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.575910367Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.576084241Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.576273024Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.576412497Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.576704713Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.576937737Z namespaces/default/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.576970848Z namespaces/default/default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577049349Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.5770824Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577161841Z namespaces/default/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577194412Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577284624Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577362885Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577444247Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577492148Z namespaces/default/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577533489Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57759989Z namespaces/default/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577644691Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577740652Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577813164Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577860095Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.577939147Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578027338Z namespaces/default/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578060919Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578158781Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578279613Z namespaces/default/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578544328Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57866126Z namespaces/default/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578744552Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.578869014Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579027817Z namespaces/default/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579094969Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579126719Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579205241Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579249632Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579332853Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579373514Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579460106Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579537197Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579601808Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579651029Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.57970709Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579743961Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579791272Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579799802Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579829633Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.579978586Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580170149Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580305102Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580508296Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580702089Z namespaces/default/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58072398Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580782591Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580824712Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580862723Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580871473Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580881823Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580921754Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580981655Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.580990895Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581044336Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581145158Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581213719Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58122161Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58127602Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581367952Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581419953Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581466724Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581571986Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581617897Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581628937Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581695608Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.581837521Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582013485Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582146337Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582335251Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582585956Z namespaces/kserve/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582623606Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582728719Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582776769Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582856291Z namespaces/kserve/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582890062Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.582980153Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583206048Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583434732Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583496073Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583573095Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583650396Z namespaces/kserve/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583682087Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583805839Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58387248Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583910741Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.583991143Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584038124Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584054114Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584094635Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584181956Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584268298Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58438162Z namespaces/kserve/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584413371Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584701426Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.584812939Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.585113434Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.585192646Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.585447131Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.585580093Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.585825968Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.5859217Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58594901Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586062712Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586090243Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586167625Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586195215Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586288967Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586367688Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.5864332Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58646053Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586533561Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586593563Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586769986Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586810007Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586848868Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586912139Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.586922159Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58697059Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587120253Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587299366Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587433969Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587654203Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587872977Z namespaces/kserve/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587885527Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.587912888Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/kserve-controller-manager-6b8548bc6d-7b946.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58801318Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58802679Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58803088Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588091262Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588213544Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588285915Z namespaces/kserve/pods/kserve-controller-manager-6b8548bc6d-7b946/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588323976Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588383277Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/llmisvc-controller-manager-7cd64959bc-vgm6v.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588490749Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588502859Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.58850853Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.588565931Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589176622Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589236073Z namespaces/kserve/pods/llmisvc-controller-manager-7cd64959bc-vgm6v/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589278754Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589321085Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/model-serving-api-fd65d7d6-vl2hg.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589418417Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/server/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589428507Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/server/server/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589440477Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/server/server/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589494949Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/server/server/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589604111Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/server/server/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589663642Z namespaces/kserve/pods/model-serving-api-fd65d7d6-vl2hg/server/server/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589681782Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589735883Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/odh-model-controller-84b54b7f97-77l6h.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589814495Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589826405Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589830595Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.589878346Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634400492Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634435322Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-77l6h/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634485893Z namespaces/kserve/pods/s3-init-5n4x5/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634538384Z namespaces/kserve/pods/s3-init-5n4x5/s3-init-5n4x5.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634666857Z namespaces/kserve/pods/s3-init-5n4x5/download-hf-model/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634677987Z namespaces/kserve/pods/s3-init-5n4x5/download-hf-model/download-hf-model/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634683247Z namespaces/kserve/pods/s3-init-5n4x5/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634701507Z namespaces/kserve/pods/s3-init-5n4x5/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634806709Z namespaces/kserve/pods/s3-init-5n4x5/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634891011Z namespaces/kserve/pods/s3-init-5n4x5/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634946672Z namespaces/kserve/pods/s3-init-5n4x5/s3-init/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634955402Z namespaces/kserve/pods/s3-init-5n4x5/s3-init/s3-init/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.634963383Z namespaces/kserve/pods/s3-init-5n4x5/s3-init/s3-init/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635047104Z namespaces/kserve/pods/s3-init-5n4x5/s3-init/s3-init/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.63537374Z namespaces/kserve/pods/s3-init-5n4x5/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635444282Z namespaces/kserve/pods/s3-init-5n4x5/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635486722Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635502383Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs-64568bcd49-g9kxn.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635626025Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635638015Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs/seaweedfs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635647785Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635711847Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.63589121Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635967052Z namespaces/kserve/pods/seaweedfs-64568bcd49-g9kxn/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.635984922Z namespaces/kserve/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636033873Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636107924Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636142595Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636281508Z namespaces/kuadrant-system/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636337029Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.63641609Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636430831Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636495312Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636535293Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.636643085Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.63692675Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637201025Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637256856Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637305057Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637360458Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.63741755Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637491521Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637557272Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637600943Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637682605Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637730996Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637740516Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637774826Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637859628Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.637940669Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638026781Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638076012Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638103563Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638280866Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638404589Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638828367Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.638908618Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639231294Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639353787Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639584841Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639716044Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639757624Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639876927Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639921098Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.639982599Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.64002569Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640123562Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640201983Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640240164Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640272445Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640340506Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640375086Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640449128Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640462078Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640511519Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640718703Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.640949927Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.64109289Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.641294474Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.64160019Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.641632601Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-5f7ps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.641945326Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.641985237Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642080759Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642160901Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642244692Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642305384Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642337604Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642412306Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642454976Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642572558Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642729422Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642822113Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642909945Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642919215Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.642943906Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino-7c77957575-czzst.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.643017507Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.643026757Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.643036078Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.643081328Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649356359Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649425721Z namespaces/kuadrant-system/pods/authorino-7c77957575-czzst/authorino/authorino/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649482262Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649510252Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/authorino-operator-6d85f6564-tsj82.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649647225Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649659835Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649667585Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649706726Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649886989Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.649957491Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-tsj82/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650013472Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650046722Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/dns-operator-controller-manager-65b49595d7-rc99j.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650163204Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650173335Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650181185Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650241486Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650357858Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650425589Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-rc99j/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.65046747Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650505601Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin-85fd4b8b9d-rqdm2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650598523Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650611013Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650618463Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650641834Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650716265Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650784927Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-rqdm2/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650803727Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650857908Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/kuadrant-operator-controller-manager-bb7ddb7d7-z679n.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.650997791Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.651009091Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.651016871Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.651047752Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667406486Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667433256Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-z679n/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667510478Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.66764098Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador-limitador-69574b596d-t6fxz.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667739932Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667750542Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667756473Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667792723Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667880175Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.667964347Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-t6fxz/limitador/limitador/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668024278Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668084949Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/limitador-operator-controller-manager-6f9f468797-xh7g7.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668217211Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668225922Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668230772Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668266132Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668440546Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668502917Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-xh7g7/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668532318Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668600219Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.66864435Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668679291Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668721011Z namespaces/kube-node-lease/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668728681Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668739582Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668806573Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-132-16.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668898545Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-141-118.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.668980306Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-141-44.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669026157Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669041167Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669088878Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669272582Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669485576Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669646759Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.669880734Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670097928Z namespaces/kube-public/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670105458Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670113148Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670163609Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670310422Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670483485Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670642198Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.670842962Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671093957Z namespaces/kube-system/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671141678Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671197329Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67124361Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671294901Z namespaces/kube-system/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671325581Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671463784Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671534855Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671654878Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671687468Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671723679Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67178169Z namespaces/kube-system/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671807391Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671897092Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671939653Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.671976544Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672072046Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672116426Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672126967Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672184558Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-xb3ky2yh2wslera473bca2tjei.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67227728Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672380791Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672479153Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672634416Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672723318Z namespaces/kube-system/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672766379Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.672968523Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.673057575Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.673372411Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.673471123Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674049184Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674167816Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674474792Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674562273Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674616985Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674677066Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674693066Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674759477Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674793648Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67488387Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.674946221Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675007682Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675061193Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675118334Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675174965Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675241757Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675251177Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675319918Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675485591Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675686535Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.675822128Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676115383Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676325578Z namespaces/kube-system/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676340538Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676390589Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer-bzwv4.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.6764697Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67647719Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676481101Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676529341Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676796576Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676861848Z namespaces/kube-system/pods/global-pull-secret-syncer-bzwv4/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676889068Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.676935459Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer-rwpm5.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677036571Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677045071Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677050001Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677078292Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677353347Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677397868Z namespaces/kube-system/pods/global-pull-secret-syncer-rwpm5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677411068Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67747702Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer-s45x2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677536801Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677556771Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677564411Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677628352Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677875407Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.677941598Z namespaces/kube-system/pods/global-pull-secret-syncer-s45x2/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67801432Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.67803452Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent-bpw6z.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.678139272Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.678155053Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.678163233Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.678210394Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.678929528Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679002179Z namespaces/kube-system/pods/konnectivity-agent-bpw6z/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679041679Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679095071Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent-d6l4b.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679168782Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679176142Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679179912Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679205043Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679913846Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.679977588Z namespaces/kube-system/pods/konnectivity-agent-d6l4b/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680016278Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680082559Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent-lcjp2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680188072Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680196152Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680200102Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680244503Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.680939056Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681014537Z namespaces/kube-system/pods/konnectivity-agent-lcjp2/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681067769Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681095089Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681186441Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/haproxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681193321Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681198131Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681243032Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681487337Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681594589Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-16.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68164633Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68168134Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681767372Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/haproxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681777332Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681781302Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.681849274Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682053857Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682114059Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-118.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68217667Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682254511Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682266052Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/haproxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682269992Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682273732Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682283512Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682603968Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.6826756Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68270763Z namespaces/kube-system/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682749151Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682804162Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682812832Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682855433Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.682953555Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683009196Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683047407Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683109568Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683136778Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683187999Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68319525Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683201579Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683265281Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683422644Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683620878Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68375851Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.683957734Z namespaces/open-cluster-management-561d7479-5fb9-4171-86d4-486c1d7a8/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684163118Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684170778Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684174888Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684210289Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684303231Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684341161Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684380482Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684424553Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684609106Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68478834Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.684928373Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685114726Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685334331Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685368621Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685449183Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685492104Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685586656Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685628146Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685713858Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68579726Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685873521Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685925572Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.685941852Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686000633Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686032604Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686118606Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686160507Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686207627Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686288379Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68634723Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.686378841Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68737127Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.687459351Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.687559683Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.687650255Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.687730777Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.687870669Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688065243Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688204336Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688239807Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688299177Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688330908Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68839815Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68842672Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688534272Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688634434Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688685065Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688714285Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688794337Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688822858Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688888459Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.68891716Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688984331Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.688992431Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.689035442Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.691401417Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.691625712Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.691763324Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.691968958Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692273334Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692285094Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692290044Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692294604Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692384606Z namespaces/openshift-apiserver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692428767Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692490108Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692539319Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692612851Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692643971Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692737203Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692814254Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692898906Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692930726Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.692981147Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693037309Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.69308084Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693160761Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693218432Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693250823Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693340265Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693427196Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693470927Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.69360297Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693683351Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693773033Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693849134Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.693929806Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694050918Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694206341Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694259462Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694302833Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694364594Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694478046Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694536877Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694608389Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694728161Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694826053Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694885094Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694918355Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.694985936Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695020797Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695072598Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695081938Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695145209Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695398584Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695608908Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695749471Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.695961955Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696183059Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.6962258Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696282291Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696314091Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696721149Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.69673671Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.69674166Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696800731Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696886302Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696897603Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.696984424Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697128567Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697313451Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697452293Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697710329Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697941163Z namespaces/openshift-authentication/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697949493Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697953753Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.697991394Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698161667Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698381421Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698547605Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698762509Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698951402Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698962213Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.698966362Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699011983Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699086045Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699101225Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699147346Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699295999Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699514343Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.699693696Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.69989316Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700111015Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700119445Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700123545Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700164706Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700223567Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700230537Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700279988Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700342149Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700349229Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7003945Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700579493Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.700758027Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70089467Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701097923Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701310158Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701380849Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70141982Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70146009Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701532172Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701609853Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701682085Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701764186Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701843808Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701885369Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.701922069Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70197458Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702007891Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702098273Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702130683Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702180084Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702261716Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702293326Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702303637Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702358518Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70245182Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702499081Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702625413Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702707734Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702788136Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702865108Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.702947169Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703087322Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703191404Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703225515Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703276405Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703331317Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703361867Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703413788Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703447659Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703536181Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703605722Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703691753Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703771335Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703826086Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703861667Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703910688Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703945218Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.703992339Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.704001159Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70403926Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.704342846Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7045428Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.704774894Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.704977938Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705166122Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705198922Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705251193Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705260794Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705286614Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705358056Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705389476Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705447547Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705481308Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70560459Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705643851Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705740623Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705770753Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705851165Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.705880695Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706015858Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706092239Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706175211Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706210672Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706249873Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706312384Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706340744Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706464627Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706539008Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706590829Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706679031Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706738342Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706745362Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706787633Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706874845Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.706959866Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.707078278Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70717479Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.707220801Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.707331643Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.707404345Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.707773312Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.707856733Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708076628Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70821351Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708329642Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708381334Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708410934Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708473215Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708513026Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708585318Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708631978Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708662389Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.70871123Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708808582Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708881113Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708916624Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708957195Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.708990765Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709044766Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709116248Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709129998Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709206959Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709376533Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709574497Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709725669Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.709913613Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710116597Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710124657Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710170998Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/aws-ebs-csi-driver-node-mkx4v.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71028409Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-driver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71029136Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-driver/csi-driver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71029542Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-driver/csi-driver/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710345631Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710442763Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710554065Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710597516Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-liveness-probe/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710604456Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710610777Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710655467Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710753609Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.710833241Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711027944Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-node-driver-registrar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711036445Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711046045Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711092136Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711191377Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711257669Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mkx4v/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71131202Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711356331Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/aws-ebs-csi-driver-node-mq2h2.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711445992Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-driver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711454082Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-driver/csi-driver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711463853Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-driver/csi-driver/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711534884Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711626486Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711706607Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711741048Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-liveness-probe/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711746218Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711752538Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711802479Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711887821Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711960302Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711985033Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-node-driver-registrar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711989783Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.711999383Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712063914Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712160236Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712229348Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-mq2h2/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712288699Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71233047Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/aws-ebs-csi-driver-node-v2rjm.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712418601Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-driver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712425641Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-driver/csi-driver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712434641Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-driver/csi-driver/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712484993Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712679306Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712750818Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712803179Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-liveness-probe/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712811209Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712815249Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71285798Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.712949071Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713019183Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713067084Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-node-driver-registrar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713074914Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713082664Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713126185Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713217156Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713285798Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-v2rjm/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713341929Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713363569Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713429341Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713437811Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713484732Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713568333Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713623624Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713694136Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713731806Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713784858Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713792427Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713798658Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713846309Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71391746Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71392923Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.713960301Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.714014692Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.714027672Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.714077503Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.714236286Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71441511Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.714576293Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.714796547Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715009781Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715039102Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715157374Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715167834Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715244626Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715263186Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715426789Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71550484Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715616873Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715662394Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715707314Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715789916Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715834417Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715884748Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.715929699Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7160121Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716073601Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716110722Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716195374Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716246065Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716261375Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716303156Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716393877Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.716438068Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.717499059Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.717640632Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.717801175Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.717881986Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71808081Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718243683Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718402516Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718572239Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7186205Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718684641Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718722912Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718780753Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718823554Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718896345Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.718924346Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719015208Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7190965Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719156361Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719208551Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719270763Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719327774Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719389055Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719434826Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719487967Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719546208Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719604949Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.71962638Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719687501Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.719866654Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720075948Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720222011Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720451095Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720777742Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720806472Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720862674Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned-fp6j9.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720989006Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.720998486Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned/tuned/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721004406Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned/tuned/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721045687Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned/tuned/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721165729Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721235221Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-fp6j9/tuned/tuned/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721393864Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721428404Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned-nzwg4.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721533466Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721548736Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned/tuned/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721552707Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned/tuned/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721563977Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned/tuned/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721684989Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721757331Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-nzwg4/tuned/tuned/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721775591Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721827202Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned-pggzv.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721911414Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721919544Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned/tuned/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721923464Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned/tuned/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.721954654Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned/tuned/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722068917Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722133608Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-pggzv/tuned/tuned/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722160618Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72221728Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72225819Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722300751Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722347632Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722358102Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722406703Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-132-16.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722505755Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-141-118.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722610737Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-141-44.ec2.internal.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722662238Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722699019Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722800251Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722845021Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722920263Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.722962164Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723025565Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723055956Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723145247Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72325848Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723357431Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723418432Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723457483Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723506794Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723576486Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723656017Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723714728Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723752089Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723836691Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723912872Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.723958983Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724145697Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724227178Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724371741Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724451992Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724590165Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724708957Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724880671Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.724932612Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725013393Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725072654Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725104795Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725189097Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725235007Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725320369Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725394321Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725434271Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725474312Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725513613Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725582044Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725683356Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725715547Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725771908Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725810858Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.725850179Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72585999Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72589326Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726051313Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726238897Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726377049Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726624894Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726815318Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726822928Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726866199Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-65db858894-jtvnl.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726959641Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-watch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726970171Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.726975911Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727018182Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727113214Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727187065Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727224726Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727230476Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727236866Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.727283027Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72796762Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728039471Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-65db858894-jtvnl/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728083582Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728137183Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728198944Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728227635Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728329577Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728361038Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728432109Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728467Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728544681Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728691854Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728771705Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.728881428Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7289946Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729052561Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729083491Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729483639Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.72953958Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729632402Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729661763Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729717234Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729798115Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729832546Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729842916Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729903177Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.729993109Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730069711Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730122621Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730222383Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730314195Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730677992Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730756894Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.730869806Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731015209Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731155411Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731218862Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731251403Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731326315Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731375625Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731448927Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731503218Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73162051Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731677781Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731763243Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731841394Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731906596Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.731935526Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732016998Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732043958Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732167621Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732183331Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732254482Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732295143Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732372565Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732408715Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732479697Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732705221Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.732928545Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733092898Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733310643Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73369482Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73371277Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733728101Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator-78495597f6-255cj.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733813532Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733821023Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733825332Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.733930625Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734071747Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734184339Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-78495597f6-255cj/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734239891Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734299092Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734337653Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734346982Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734413004Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734484185Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734570777Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734677069Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73472743Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734815132Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734871223Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734940564Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.734983685Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735030926Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735055266Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735150778Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73523475Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735356462Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735403363Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735453384Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735510285Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735566196Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735657908Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735723159Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73576895Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735849131Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735907943Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735915013Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.735962064Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736069226Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736085036Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736203938Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73628217Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736381652Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736461463Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736569475Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736700438Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736892671Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736949043Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.736990853Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737056695Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737099115Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737129576Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737189417Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737265289Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73734893Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737386361Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737422642Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737492143Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737532964Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737697457Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737708167Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737743698Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.737979552Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738187046Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738321219Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738511633Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738727877Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738758497Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738827809Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738856409Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.738961331Z namespaces/openshift-config-managed/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739011732Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739085954Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739131374Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739196246Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739238527Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739322598Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73939941Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739481981Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739532662Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739646995Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739698265Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739725666Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739815678Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.739889959Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.73992857Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.740009551Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.740096473Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.740139884Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.743619911Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.743705793Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.743795304Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.743881146Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.743966307Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744191262Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744298464Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744352825Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744382995Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744482517Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744571929Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74459197Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744659291Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744700261Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744765383Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744795814Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744890455Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.744961997Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745008547Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745036568Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745100779Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7451457Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745202511Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745212101Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745264912Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745427695Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745639059Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745780742Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.745990786Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746227191Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746269792Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746323803Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746332333Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746370314Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746443525Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746477436Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746567348Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746613488Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746664389Z namespaces/openshift-config-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74667289Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74668077Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746723151Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746797822Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746808192Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.746852603Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.747012026Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74720778Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.747353843Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.747678089Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.747891813Z namespaces/openshift-config/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.747940614Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748221419Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74826736Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748344962Z namespaces/openshift-config/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748386583Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748471414Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748578056Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748664328Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748715029Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74875073Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748818471Z namespaces/openshift-config/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748854381Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748941533Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.748987264Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749031055Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749113306Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749164937Z namespaces/openshift-config/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749201038Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74932315Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749399462Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749482384Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749574505Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749661167Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.74980826Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749945112Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.749999123Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750033554Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750092205Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750139476Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750193897Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750239588Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750291619Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75034355Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750416341Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750504323Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750567834Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750621695Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750685667Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750740358Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750775628Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750785438Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.750841499Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751007133Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751184676Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751320309Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751534573Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751730467Z namespaces/openshift-config/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751757867Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751817088Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751844019Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.751960661Z namespaces/openshift-console-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752009682Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752093024Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752126874Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752214766Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752249677Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752338088Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752453061Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752588443Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752618004Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752672255Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752720926Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752764717Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752844908Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.752878069Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75292914Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753011091Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753043812Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753050522Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753083383Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753163734Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753192515Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.753903118Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75398897Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754243925Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754319337Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754445449Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754604532Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754785555Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754843507Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754893608Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754943878Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.754976309Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75504232Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755082451Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755165873Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755245014Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755309105Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755341216Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755403427Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755436578Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755500699Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75555083Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755624931Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755645512Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755711573Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755729993Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755773344Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.755931657Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756120471Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756249834Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756441347Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756659222Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756671212Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756705052Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/console-operator-7866497494-vt7q5.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756903416Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/console-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756911416Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/console-operator/console-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756915346Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/console-operator/console-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.756960477Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/console-operator/console-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757552629Z namespaces/openshift-console-operator/pods/console-operator-7866497494-vt7q5/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75759796Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757657151Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757713862Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757744002Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757811974Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757820524Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757832924Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.757874835Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.758035028Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.758211651Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.758345574Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.758559788Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.758774082Z namespaces/openshift-console/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.758824073Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759021547Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759052708Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759131579Z namespaces/openshift-console/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.75917792Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759259612Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759427265Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759741611Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759779011Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759818692Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759885424Z namespaces/openshift-console/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.759925094Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760007636Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760074487Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760120048Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760199059Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760259501Z namespaces/openshift-console/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760287051Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.760988465Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.761074166Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.761343472Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.761419053Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.761657038Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76178532Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.761967324Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762026645Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762071466Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762136177Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762175137Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762240279Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76228665Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762364251Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762449813Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762505424Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762643966Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762696947Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762738968Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762800709Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76282893Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762897211Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762907282Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.762956363Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.763106285Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.763283189Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.763425152Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.763653576Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76384824Z namespaces/openshift-console/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76385787Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.763908991Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console-6bdf4df5b-jxd89.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764025533Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764033133Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console/console/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764037093Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console/console/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764089194Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console/console/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764188936Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console/console/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764259528Z namespaces/openshift-console/pods/console-6bdf4df5b-jxd89/console/console/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764294548Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764321699Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/downloads-68c9cbb497-54pcn.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764422131Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/download-server/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764429511Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/download-server/download-server/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764433511Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/download-server/download-server/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764452691Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/download-server/download-server/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764831068Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76490153Z namespaces/openshift-console/pods/downloads-68c9cbb497-54pcn/download-server/download-server/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76492731Z namespaces/openshift-console/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.764973471Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765032832Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765065603Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765186395Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765235306Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765293727Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765333618Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765383969Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76542395Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765506092Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765603683Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765685005Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765730466Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765764826Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765833618Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765865358Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.765984251Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766042152Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766074272Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766164744Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766406459Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76645221Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766585722Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766664364Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766742815Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766827957Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.766902888Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767022541Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767182934Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767235725Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767284516Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767339307Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767386758Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767452089Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76748425Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767598742Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767678443Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767725684Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767762165Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767812376Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767846946Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767896178Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767903118Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.767939958Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768107571Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768294775Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768432288Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768652232Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768837245Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768864956Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768929017Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.768955368Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7690456Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.76908225Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769142621Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769172152Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769231113Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769271254Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769356795Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769440947Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769530999Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7696015Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769643741Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769703032Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769737523Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769827075Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769874636Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.769921906Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770000008Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770057999Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770065109Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77009563Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770195812Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770225162Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770343754Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770420396Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770513658Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77066028Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770746092Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.770870604Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771034138Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771101789Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771188031Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771235932Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771267572Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771323053Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771364634Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771446856Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771576798Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77165293Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771699851Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771743081Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771786072Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771834463Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771841423Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.771876174Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.772108448Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.772491516Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.772690099Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.772948135Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773331972Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773369493Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773425174Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773469054Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773598317Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773651338Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773712609Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77374579Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773813511Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773851522Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.773939514Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774018545Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774101057Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774159468Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774193688Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774247549Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77427584Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774375032Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774433933Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774464314Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774590186Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774668198Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774710398Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774820651Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.774902242Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775006854Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775086446Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775171017Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77530086Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775488823Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775595955Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775641086Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775697477Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775744078Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775795369Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77583905Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775920132Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.775999843Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776035484Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776103495Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776158156Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776184157Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776254698Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776303189Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77633715Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776397511Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776440102Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776447992Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776489033Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.776687436Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77686711Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777004702Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777200716Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77739175Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77742299Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777489602Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777532212Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777639685Z namespaces/openshift-dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777695336Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777752097Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777780767Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77791165Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.777958461Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778105054Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778181805Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778262877Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778305267Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778332748Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778408049Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77844427Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778536302Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778698755Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778744096Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778828168Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.778894239Z namespaces/openshift-dns/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77894012Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.779045162Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.779131734Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.77949521Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.779611433Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.779892738Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78000879Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.780205654Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.780917748Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.780967339Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78104357Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781072311Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781149412Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781174383Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781269344Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781345996Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781436778Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781483659Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781534509Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781591831Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781652392Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781692513Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781727413Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781737004Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781788965Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.781931197Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782105251Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782241723Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782440407Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782674362Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782687972Z namespaces/openshift-dns/pods/dns-default-82fgg/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782736043Z namespaces/openshift-dns/pods/dns-default-82fgg/dns-default-82fgg.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782829014Z namespaces/openshift-dns/pods/dns-default-82fgg/dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782836305Z namespaces/openshift-dns/pods/dns-default-82fgg/dns/dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782846745Z namespaces/openshift-dns/pods/dns-default-82fgg/dns/dns/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782895046Z namespaces/openshift-dns/pods/dns-default-82fgg/dns/dns/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.782993848Z namespaces/openshift-dns/pods/dns-default-82fgg/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783061729Z namespaces/openshift-dns/pods/dns-default-82fgg/dns/dns/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7831028Z namespaces/openshift-dns/pods/dns-default-82fgg/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7831097Z namespaces/openshift-dns/pods/dns-default-82fgg/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78311734Z namespaces/openshift-dns/pods/dns-default-82fgg/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783165491Z namespaces/openshift-dns/pods/dns-default-82fgg/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783259143Z namespaces/openshift-dns/pods/dns-default-82fgg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783326374Z namespaces/openshift-dns/pods/dns-default-82fgg/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783351755Z namespaces/openshift-dns/pods/dns-default-d4dmb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783399575Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns-default-d4dmb.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783481287Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783488677Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns/dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783498067Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns/dns/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783565288Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns/dns/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783670631Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783743262Z namespaces/openshift-dns/pods/dns-default-d4dmb/dns/dns/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783789213Z namespaces/openshift-dns/pods/dns-default-d4dmb/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783796703Z namespaces/openshift-dns/pods/dns-default-d4dmb/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783803063Z namespaces/openshift-dns/pods/dns-default-d4dmb/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783849424Z namespaces/openshift-dns/pods/dns-default-d4dmb/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.783944176Z namespaces/openshift-dns/pods/dns-default-d4dmb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784011447Z namespaces/openshift-dns/pods/dns-default-d4dmb/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784087949Z namespaces/openshift-dns/pods/dns-default-dzsz9/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784116779Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns-default-dzsz9.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784190731Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784199111Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns/dns/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784207761Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns/dns/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784261422Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns/dns/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784351814Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784421405Z namespaces/openshift-dns/pods/dns-default-dzsz9/dns/dns/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784448886Z namespaces/openshift-dns/pods/dns-default-dzsz9/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784456546Z namespaces/openshift-dns/pods/dns-default-dzsz9/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784462656Z namespaces/openshift-dns/pods/dns-default-dzsz9/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784536937Z namespaces/openshift-dns/pods/dns-default-dzsz9/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784643349Z namespaces/openshift-dns/pods/dns-default-dzsz9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784778122Z namespaces/openshift-dns/pods/dns-default-dzsz9/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784805623Z namespaces/openshift-dns/pods/node-resolver-8j9n9/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784850743Z namespaces/openshift-dns/pods/node-resolver-8j9n9/node-resolver-8j9n9.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784939165Z namespaces/openshift-dns/pods/node-resolver-8j9n9/dns-node-resolver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784946635Z namespaces/openshift-dns/pods/node-resolver-8j9n9/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.784956005Z namespaces/openshift-dns/pods/node-resolver-8j9n9/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785001286Z namespaces/openshift-dns/pods/node-resolver-8j9n9/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785087888Z namespaces/openshift-dns/pods/node-resolver-8j9n9/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785155839Z namespaces/openshift-dns/pods/node-resolver-8j9n9/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78521201Z namespaces/openshift-dns/pods/node-resolver-dd2pj/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785250271Z namespaces/openshift-dns/pods/node-resolver-dd2pj/node-resolver-dd2pj.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785346743Z namespaces/openshift-dns/pods/node-resolver-dd2pj/dns-node-resolver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785353953Z namespaces/openshift-dns/pods/node-resolver-dd2pj/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785357953Z namespaces/openshift-dns/pods/node-resolver-dd2pj/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785392394Z namespaces/openshift-dns/pods/node-resolver-dd2pj/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785484625Z namespaces/openshift-dns/pods/node-resolver-dd2pj/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785581907Z namespaces/openshift-dns/pods/node-resolver-dd2pj/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785600218Z namespaces/openshift-dns/pods/node-resolver-z9spw/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785653189Z namespaces/openshift-dns/pods/node-resolver-z9spw/node-resolver-z9spw.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78572712Z namespaces/openshift-dns/pods/node-resolver-z9spw/dns-node-resolver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78573411Z namespaces/openshift-dns/pods/node-resolver-z9spw/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78573791Z namespaces/openshift-dns/pods/node-resolver-z9spw/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785766431Z namespaces/openshift-dns/pods/node-resolver-z9spw/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785850832Z namespaces/openshift-dns/pods/node-resolver-z9spw/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785920384Z namespaces/openshift-dns/pods/node-resolver-z9spw/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785947244Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.785979035Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786034676Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786080807Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786169039Z namespaces/openshift-etcd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78621111Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786274131Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786307691Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786373433Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786405533Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786496825Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786712019Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786791491Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.786835031Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787015355Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787023055Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787027755Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787036095Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787099127Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787141587Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787227029Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78730536Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787337511Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787437813Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787528855Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787613876Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787691868Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787839301Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.787957863Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788128616Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788194628Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788231438Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788284539Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78832707Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788374061Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788411232Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788498563Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788592885Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788652236Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788697767Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788744588Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788786439Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78882321Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78883314Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.788888871Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.789053934Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.789232188Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.7893696Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.789599255Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.789797519Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.789840019Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.78989808Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.789929261Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790036203Z namespaces/openshift-host-network/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790073054Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790152055Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790183396Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790248527Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790279588Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790371979Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790449151Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790529613Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790603834Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790645125Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790703346Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790750107Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790826878Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.790885639Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79091397Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791006002Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791081903Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791129984Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791226316Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791304957Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791382409Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79146284Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791553212Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791704815Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791806757Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791855538Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791905629Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79196709Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.791998501Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792073612Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792124393Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792186124Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792217625Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792308157Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792386948Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792433859Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79246502Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792533751Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792578002Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792616902Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792638993Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792692934Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.792842067Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793036271Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793173783Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793372907Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793583011Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793628942Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793674793Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793718074Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793797315Z namespaces/openshift-image-registry/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793841206Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793902457Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793933988Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.793987939Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794017469Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794226664Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794361836Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79458498Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794630521Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794665622Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794716603Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794759484Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794858606Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794918377Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.794948527Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.795045679Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79509218Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79510057Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.795125301Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.795209943Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.795247213Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.796301433Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.796388275Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79665187Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.796733962Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.796943366Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797070048Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797273332Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797314133Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797353853Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797430655Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797468476Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797500136Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797605969Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79768592Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797773602Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797831793Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797864493Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797939065Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.797982346Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798095508Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79818667Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798254891Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798298882Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798348743Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798396554Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798440834Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798454435Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798513586Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798790281Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.798970925Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799129958Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799344142Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799568636Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799581566Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799608717Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/image-registry-7dbc88fcb8-4rjtv.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799717379Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/registry/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799724539Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/registry/registry/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.799730779Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/registry/registry/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.79978078Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/registry/registry/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801163367Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801231898Z namespaces/openshift-image-registry/pods/image-registry-7dbc88fcb8-4rjtv/registry/registry/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801248968Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801302229Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca-5qmgm.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801391341Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801401591Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca/node-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801406642Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca/node-ca/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801416992Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca/node-ca/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801613395Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801665287Z namespaces/openshift-image-registry/pods/node-ca-5qmgm/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801697477Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801739808Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca-hdxkr.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80182227Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80183481Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca/node-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80184719Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca/node-ca/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.801896691Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca/node-ca/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802040914Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802111045Z namespaces/openshift-image-registry/pods/node-ca-hdxkr/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802125495Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802176206Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca-m9ntl.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802248838Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802256698Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca/node-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802266198Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca/node-ca/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802330309Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca/node-ca/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802472722Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802540803Z namespaces/openshift-image-registry/pods/node-ca-m9ntl/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802600514Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802616075Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802713096Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802729727Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802789538Z namespaces/openshift-infra/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802799198Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802806758Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802819729Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.802987662Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.803160845Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.803295778Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.803497732Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.803842308Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.803879219Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80396002Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804001771Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804074363Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804124674Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804234736Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804313577Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804395579Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80445262Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804496411Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804590083Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804629343Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804717815Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804747426Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804799597Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.804881488Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80496342Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80499878Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805113863Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805199764Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805362507Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805442289Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805637343Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805770795Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.805937589Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806060641Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806094712Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806162493Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806193003Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806271215Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806300916Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806393727Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806471749Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80653041Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806580271Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806627792Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806660332Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806724354Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806745364Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806787565Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.806942278Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.807118821Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.807381006Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.807618741Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.807873836Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.807881746Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.807927957Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/ingress-canary-6cwhv.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808009889Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/serve-healthcheck-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808021939Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808034709Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80808173Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808176102Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808244433Z namespaces/openshift-ingress-canary/pods/ingress-canary-6cwhv/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808305244Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808324284Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/ingress-canary-8qn5m.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808425217Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/serve-healthcheck-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808433837Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808448087Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808487158Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80861332Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808753563Z namespaces/openshift-ingress-canary/pods/ingress-canary-8qn5m/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.808956787Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809000457Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/ingress-canary-znkdl.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809073769Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/serve-healthcheck-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809085089Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809091499Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80914087Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809238922Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809306923Z namespaces/openshift-ingress-canary/pods/ingress-canary-znkdl/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809344544Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809382905Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809440926Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809484747Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809617769Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.80966344Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809740482Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809781712Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809846964Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809889505Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.809974596Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810051928Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810134199Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81018782Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810233701Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810300302Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810345943Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810424255Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810474356Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810513326Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810624029Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81069889Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.810744841Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.811333482Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.811416164Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.811537356Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.811650528Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81173273Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.811856182Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812063936Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812122708Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812164608Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81222561Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81227154Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812320601Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812330081Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812380062Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812446394Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812491764Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812597666Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812681048Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81275296Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81280507Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812841401Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812889942Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.812972884Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813001254Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813073046Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813104796Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813174408Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813181738Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813216698Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81329936Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81331082Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813362631Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813653557Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81383513Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.813974133Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814183287Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814390461Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814432422Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814492053Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814541124Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814666706Z namespaces/openshift-ingress/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814743378Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814940772Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.814982672Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815037744Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815092265Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815175746Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815409651Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815637875Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815695266Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815741317Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815800338Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815847769Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81592359Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.815989022Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816021672Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816118164Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816174275Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816188336Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816235287Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816324178Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81640813Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816505332Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.816576603Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.817013502Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.817123374Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.817332288Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.817413559Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.817698675Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.817819027Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818034341Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818131443Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818178934Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818284896Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818291876Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818329967Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818410138Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818444049Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.81851501Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818575141Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818684724Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818754045Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818818106Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818862327Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818928968Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.818975149Z namespaces/openshift-ingress/monitoring.coreos.com/podmonitors/openshift-ai-inference-metrics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819046581Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819086291Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819141862Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819187583Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819241104Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819250585Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819307186Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819456498Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819691483Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.819832316Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.82005333Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820272414Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820279644Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820315125Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/istiod-openshift-gateway-94bb8fbfd-p9b99.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820440167Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820447597Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820456288Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.820498878Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854283538Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854334159Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-p9b99/discovery/discovery/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.85438754Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854432981Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/openshift-ai-inference-openshift-default-866675cbb8-cvzqk.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854802928Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854819028Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854824418Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.854869429Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.857855967Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-866675cbb8-cvzqk/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.857940938Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.857988579Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router-default-5bc67f59d-8gfgw.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858113701Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858122152Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858133042Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858183733Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858305785Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858347986Z namespaces/openshift-ingress/pods/router-default-5bc67f59d-8gfgw/router/router/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858389687Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858446618Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858505629Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.85857244Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858672552Z namespaces/openshift-insights/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858723433Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858802835Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858831245Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858915677Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.858933737Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.85906905Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859261423Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859361405Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859409206Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859448627Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859545159Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.85957889Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859647961Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859679342Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859767853Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859824164Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859871795Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.859953257Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.860030978Z namespaces/openshift-insights/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86009289Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.860991257Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.861113529Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.861401965Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.861543427Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.861842743Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.861935475Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862093818Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862162039Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86219941Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862266371Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862302342Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862392734Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862450465Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862483645Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862579907Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862600378Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862686259Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.8627217Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862784071Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862813042Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862909244Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.862989555Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863051196Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863088457Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863161558Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863197439Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863278611Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863295451Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863388003Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863437774Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863479755Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863491215Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863576066Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86377799Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.863985154Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.864155148Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.864351591Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86480207Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86481824Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.864926052Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/insights-operator-68d9dd94cf-ft7pk.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865032285Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/insights-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865043465Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/insights-operator/insights-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865055425Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/insights-operator/insights-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865114976Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865480463Z namespaces/openshift-insights/pods/insights-operator-68d9dd94cf-ft7pk/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865786299Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86581498Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/insights-runtime-extractor-5vv52.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865933882Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865942042Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/exporter/exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865946262Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/exporter/exporter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.865985843Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/exporter/exporter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866076394Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866171946Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/exporter/exporter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866200067Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/extractor/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866207417Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/extractor/extractor/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866211327Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/extractor/extractor/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866256678Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/extractor/extractor/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86635987Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866430101Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/extractor/extractor/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866461322Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866470712Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866484452Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866558864Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866663826Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866729657Z namespaces/openshift-insights/pods/insights-runtime-extractor-5vv52/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866772728Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866816469Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/insights-runtime-extractor-fsgn4.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86690781Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866915111Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/exporter/exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866928871Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/exporter/exporter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.866970842Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/exporter/exporter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867063573Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867131325Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/exporter/exporter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867163865Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/extractor/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867173975Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/extractor/extractor/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867178276Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/extractor/extractor/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867227777Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/extractor/extractor/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867326718Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86740157Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/extractor/extractor/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867437151Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867634584Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867650545Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867680455Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867774807Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867842668Z namespaces/openshift-insights/pods/insights-runtime-extractor-fsgn4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.867891719Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86793129Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/insights-runtime-extractor-zpv9t.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868019662Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868026762Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/exporter/exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868030802Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/exporter/exporter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868084253Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/exporter/exporter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868172925Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868246616Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/exporter/exporter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868281487Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/extractor/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868289417Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/extractor/extractor/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868294337Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/extractor/extractor/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868335678Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/extractor/extractor/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86842916Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868500371Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/extractor/extractor/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868563462Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868576022Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868580093Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868621333Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868710965Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868781726Z namespaces/openshift-insights/pods/insights-runtime-extractor-zpv9t/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868818177Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868861468Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.868914779Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.86895511Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869007141Z namespaces/openshift-keda/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869014581Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869022161Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869067732Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869156094Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869203485Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869211595Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869245035Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869311237Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869350337Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869440869Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869550891Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869608952Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869623123Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869669543Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.869821457Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87000289Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.870195814Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.870399868Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.870568691Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.870898587Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.870940798Z namespaces/openshift-keda/operators.coreos.com/installplans/install-54wct.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87105452Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871100031Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871151162Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871194703Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871255124Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871286985Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871375116Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871420487Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871481418Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871543439Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871611231Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871642452Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871741333Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871814955Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871900686Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871939187Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.871982788Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872032579Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87208396Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872163082Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872216723Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872262564Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872342195Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872397686Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872440117Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872573219Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872655691Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872744003Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872822164Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.872898806Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873020898Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873201862Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873250712Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873293103Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873346544Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873389405Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873447496Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873475547Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87365306Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873732692Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873780453Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873818953Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873862834Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873908265Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873955316Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.873962786Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.874009327Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87416643Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.874342643Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.874480116Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87469766Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.874885624Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.874926675Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.874978076Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875019546Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875092858Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875138239Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87519795Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875238901Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875292402Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875335203Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875418484Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875493766Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875598988Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875645269Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875684919Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.8757405Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875773471Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875857322Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875910614Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.875951274Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876035706Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876104957Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876142618Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87624558Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876334442Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876414463Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876494195Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876597617Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876718989Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876877852Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876930003Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.876973274Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877020305Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877061216Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877108406Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877148427Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877227609Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87731631Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877365671Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877417383Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877479184Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877542865Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877668407Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877721858Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877759799Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87781538Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877862771Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877912712Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877919992Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.877968293Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.878114066Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.878477033Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.878641416Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87884306Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879044194Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879082035Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879137105Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879181996Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879247038Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879298539Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.8793627Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879402991Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879461022Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879504232Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879609175Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879687776Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879765928Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879817938Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879846899Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.87989799Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.879936191Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880021702Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880080294Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880109834Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880201266Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880258537Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880303748Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88040829Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880490491Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880589673Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880670925Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880748267Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.880876099Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881053192Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881117674Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881145544Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881207015Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881255456Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881305367Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881347728Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881430469Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881508791Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881584783Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881616663Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881674894Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881725645Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881782426Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881827727Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881875838Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881888969Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.881949259Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.882113243Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.882301966Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.882440369Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.882750225Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.882936709Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.882967429Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88303596Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883085191Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883181523Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883229094Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883296576Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883330216Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883401767Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883441088Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88353727Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883630912Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883712173Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883772854Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883821315Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883882257Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.883915367Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884003319Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88406891Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884120661Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884200953Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884260854Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884267814Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884308105Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884395507Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884442838Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88456451Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884644721Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.884932297Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885012118Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88509337Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885227993Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885374415Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885424866Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885473687Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885513258Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88558792Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885651611Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885703462Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885780953Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885859195Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885906126Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885945776Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.885988777Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.886037338Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.886080609Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.886087839Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88612552Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.886321423Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.886502917Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88666169Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.886859274Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887040648Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887069558Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887125449Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88716584Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887250712Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887297422Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887369694Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887397214Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887468036Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887510546Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887617499Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88770139Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887786302Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887830763Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887871884Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887923494Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.887951685Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888041417Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888085357Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888137498Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88821398Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888293712Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888336072Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888446744Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888538046Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888620628Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888702299Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888788251Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.888904963Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889058596Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889107557Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889150738Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889212149Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88924131Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889320981Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889364862Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889448044Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889539305Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889583366Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889638587Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889679048Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889718299Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.88976818Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889800341Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889842921Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.889992524Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.890177958Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89031727Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.890709168Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.890906452Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.890944483Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891011314Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891054534Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891134666Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891169317Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891257098Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891296919Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89134442Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891385671Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891468042Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891566544Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891715727Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891765388Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891803079Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89186405Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891908241Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.891994273Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892035673Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892088505Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892166356Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892224517Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892258078Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.8923596Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892440051Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892540873Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892625095Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892703796Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892829669Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.892984192Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893038993Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893070523Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893124554Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893155165Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893218766Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893250047Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893337329Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89341416Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893460861Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893489271Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893567163Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893609484Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893666475Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893673235Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893717186Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.893867129Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894046972Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894184285Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894372438Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894577972Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894617133Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894669714Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894701855Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894783916Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894831287Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894891788Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.894927569Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89497151Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895004741Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895100882Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895224195Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895327127Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895374597Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895417988Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895468989Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.8955112Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895614812Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895664363Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895705444Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895785956Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895832856Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895840886Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895882277Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895946719Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.895978159Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896173453Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896255725Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896432418Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896514019Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896665442Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896782925Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.896961378Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897025459Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89705411Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897119231Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897154722Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897214293Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897242184Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897339475Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897419837Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897470068Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897512339Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897675742Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897695892Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897761394Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897772114Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897810715Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.897966147Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898158771Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898296014Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898498768Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898711862Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898726882Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898750293Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/kube-storage-version-migrator-operator-799cb7f958-rssm7.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898877115Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/kube-storage-version-migrator-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898887575Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898898795Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.898950026Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899062539Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-799cb7f958-rssm7/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899188561Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899218082Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899273632Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899316223Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899379095Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899428166Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899485477Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899531177Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899601809Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.89964797Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899736161Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899853484Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.899946525Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900005387Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900054558Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900092968Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900149709Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900228541Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900284762Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900320603Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900447745Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900505596Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900569628Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90072089Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900801912Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.900922964Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901004426Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901115688Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90124319Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901397393Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901437644Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901487665Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901561787Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901599937Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901653718Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901699019Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901806151Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901891473Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901937614Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.901985685Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902034156Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902075206Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902128027Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902135188Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902177839Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902336082Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902513415Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902677878Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.902870932Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903163707Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903173607Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903213048Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator-7f59b54f6c-rxzlx.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90330047Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/graceful-termination/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90331081Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/graceful-termination/graceful-termination/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90332051Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903356581Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903449343Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903533184Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903564625Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903569345Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator/migrator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903573045Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator/migrator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903628456Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator/migrator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903731478Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903795539Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-7f59b54f6c-rxzlx/migrator/migrator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90383373Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903872361Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903914842Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.903950793Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904017194Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904075065Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904147396Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904238648Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904302629Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90433487Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904422571Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904588405Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904738658Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904790469Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90483771Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9048833Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.904915891Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905015953Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905068184Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905109485Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905221547Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905267218Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905277078Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905315609Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90540367Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905463671Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905512263Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905704586Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.905797248Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906123464Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906275967Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906459511Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906606124Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906791617Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906856108Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.906885209Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90695825Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907008011Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907053112Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907090753Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907170924Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907253846Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907302547Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907330327Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907397909Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907430149Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907494741Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907535481Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907610443Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907621483Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907659924Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.907812047Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9079958Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.908152403Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.908380967Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90853301Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.908831276Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.908865447Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-6fk2s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.908949369Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.908979079Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90903055Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909069521Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909124942Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909154133Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909229134Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909239484Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909280435Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/lws-controller-manager-6cd758bd69-ld6ng.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909371577Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909379857Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909390527Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909412618Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.90957051Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909643062Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-ld6ng/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909676152Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909722833Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/lws-controller-manager-6cd758bd69-zwxvw.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909801345Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909808445Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/manager/manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909812875Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/manager/manager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.909844896Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/manager/manager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91008742Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910153042Z namespaces/openshift-lws-operator/pods/lws-controller-manager-6cd758bd69-zwxvw/manager/manager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910185162Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910224113Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator-fd8ccff4c-tlsc4.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910314075Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910324225Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910335225Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910372306Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910561589Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910631151Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-tlsc4/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910657931Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910703762Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910755473Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910782734Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910838795Z namespaces/openshift-machine-api/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910845765Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910854885Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.910882066Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91110121Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911281613Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911420216Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911666051Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911861685Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911872715Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911878995Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.911899795Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912004287Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912089469Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912100019Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91213566Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912294153Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912472616Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912626239Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.912819553Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913015757Z namespaces/openshift-marketplace/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913026597Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913032997Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913042177Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913124939Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913135379Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913145769Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913252051Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913333413Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913421055Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913483236Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913536157Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91369703Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.913875793Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914011206Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914201619Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914435424Z namespaces/openshift-monitoring/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914475675Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914624558Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914657888Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91473159Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914775181Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.914942694Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.915378962Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.915762669Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916096166Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916119826Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916172767Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916219728Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91629753Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916370541Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916403502Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916499434Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916562835Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.916590585Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.919817478Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.91995629Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.920903318Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.92098185Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.921921958Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.92203893Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.922515079Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.922771024Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.922809995Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.922969468Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.922991718Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923050889Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923212213Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923224473Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923260833Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923304825Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923345825Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923474718Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923512178Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923616271Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923649771Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923761953Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923780374Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.923894826Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9240927Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924178521Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924429736Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924583759Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924709352Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924799063Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924903635Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.924994187Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925070958Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925119099Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925203191Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925292273Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925379174Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925495567Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925596629Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925750861Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925842703Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.925933905Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926022687Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926108978Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9261988Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926280842Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926313952Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926372284Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926379284Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926417944Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926623238Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.926860803Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927055306Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.92726243Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927484575Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927552816Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927631257Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927638598Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927706709Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927871002Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927878142Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927882242Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.927925893Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928063776Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928139917Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928154378Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928161978Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928166638Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928215589Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928316551Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928381752Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928420373Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928427813Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928434133Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928465064Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928577046Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928647337Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928679678Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928690988Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928699468Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928733779Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928830141Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928897022Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928928293Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928936263Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928941043Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.928982484Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929076816Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929139607Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929166257Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929170907Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929176767Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.92930519Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929390782Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929462103Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929497073Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929505724Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929515514Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929582555Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929668157Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929736878Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929770419Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.92981609Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator-5f7f494dd4-rnn78.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929906871Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929914072Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929918192Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.929965922Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.930725387Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.930796258Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-5f7f494dd4-rnn78/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.930842129Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93087012Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics-785778ff74-98d5p.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.930983142Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-main/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.930990362Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.930994462Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931033133Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931128355Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931198056Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931225167Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-self/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931233427Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931237367Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931280898Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931373369Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931435051Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931471842Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931479232Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931483312Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931528572Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931736567Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931803618Z namespaces/openshift-monitoring/pods/kube-state-metrics-785778ff74-98d5p/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931841418Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931869069Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server-5975c47848-c26vd.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931969311Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931976501Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server/metrics-server/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.931980461Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server/metrics-server/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932016872Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932127734Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932197175Z namespaces/openshift-monitoring/pods/metrics-server-5975c47848-c26vd/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932233666Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932259906Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin-5f68df7f58-x7rcz.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932354758Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932361848Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932365628Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932396549Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932514902Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932596283Z namespaces/openshift-monitoring/pods/monitoring-plugin-5f68df7f58-x7rcz/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932638084Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932684265Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter-tjnk7.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932787757Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/init-textfile/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932796947Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/init-textfile/init-textfile/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932800877Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/init-textfile/init-textfile/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932830048Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.932905819Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93297169Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933005051Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933011591Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933015441Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933044252Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933144674Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933212955Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933266416Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933276706Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter/node-exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933280846Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter/node-exporter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933312237Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933422209Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93348937Z namespaces/openshift-monitoring/pods/node-exporter-tjnk7/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933534181Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933592752Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter-wl64d.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933691364Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/init-textfile/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933699444Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/init-textfile/init-textfile/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933703564Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/init-textfile/init-textfile/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933737085Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933809246Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933875598Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933904708Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933909518Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933913368Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.933958629Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934051081Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934120992Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934166983Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934174493Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter/node-exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934178433Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter/node-exporter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934208754Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934315846Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934384627Z namespaces/openshift-monitoring/pods/node-exporter-wl64d/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934432178Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934459239Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter-x6v6k.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934590731Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/init-textfile/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934601651Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/init-textfile/init-textfile/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934605611Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/init-textfile/init-textfile/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934643612Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934716594Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934782355Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934817996Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934824916Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934828706Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934873297Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.934983649Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93505393Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935095291Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935100791Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter/node-exporter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935107021Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter/node-exporter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935152212Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935270514Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935336066Z namespaces/openshift-monitoring/pods/node-exporter-x6v6k/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935393217Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935437148Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics-84bd68b896-v6nj5.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935601481Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-main/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935611721Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935617761Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935660262Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935759354Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935823265Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935872946Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-self/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935880876Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935891086Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.935932497Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936022569Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936102591Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936140041Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936146091Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936156592Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936199852Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936307954Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936376406Z namespaces/openshift-monitoring/pods/openshift-state-metrics-84bd68b896-v6nj5/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936445357Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936509258Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936720362Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936732732Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936736733Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936781533Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936888646Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936955147Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.936977887Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937007928Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937014098Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937054409Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93714289Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937210432Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937239352Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937244402Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937248082Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937291763Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937378805Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937451876Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937468637Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937474277Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937486107Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937534368Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93764328Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937711881Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937745302Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937749962Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937756002Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937801193Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937895405Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937962176Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937992827Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.937997407Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938011037Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938051618Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938479946Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938568018Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938604419Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938610639Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938614999Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93866082Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938757162Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938828273Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938861144Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.938907914Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator-64785b6675-xqhrm.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939108698Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939116568Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939120598Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939155809Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939247301Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939311372Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939347533Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939354983Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939360793Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939401214Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939655149Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93972802Z namespaces/openshift-monitoring/pods/prometheus-operator-64785b6675-xqhrm/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.93974452Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939804782Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook-77d98bc786-khplm.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939925154Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939932724Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939936754Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.939976085Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940084417Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940151638Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-77d98bc786-khplm/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940190049Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94022858Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client-5449c784b4-nglgv.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940396363Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940410523Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940418433Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940434894Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940548706Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940612157Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940629087Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/reload/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940636658Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/reload/reload/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940659748Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/reload/reload/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940715519Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/reload/reload/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940803781Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940874132Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/reload/reload/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940899273Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940904043Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client/telemeter-client/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940907673Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.940955494Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941054006Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941118377Z namespaces/openshift-monitoring/pods/telemeter-client-5449c784b4-nglgv/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941135837Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941205009Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-querier-6779965f77-zwksh.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941362852Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941371542Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941375812Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941417363Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941510935Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941623126Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941713188Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-rules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941721999Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941733139Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941768019Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941863491Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941934223Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941966883Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-web/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941977273Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.941985824Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942022034Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942149957Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942220598Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942250189Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942254679Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942260729Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94230693Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942395211Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942469323Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942503443Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/prom-label-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942508134Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942512203Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942562724Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942668907Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942732488Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942777909Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-query/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942785429Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-query/thanos-query/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942789349Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-query/thanos-query/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942816789Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942920362Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.942988103Z namespaces/openshift-monitoring/pods/thanos-querier-6779965f77-zwksh/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943024833Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943053624Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943120065Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943148576Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943260338Z namespaces/openshift-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943327629Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943398291Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943436091Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943479292Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943540363Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943761978Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943839909Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943928041Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.943972832Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944017683Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944175766Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944291048Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944354449Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94440189Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94443786Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944473961Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944586723Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944638755Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944761017Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944842428Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94491025Z namespaces/openshift-multus/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.944941Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.945080563Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.945168635Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.945693805Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.945802847Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.946544831Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.946642343Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.946769296Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.946819957Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.946867017Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.946964599Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947050041Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947127872Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947221714Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947251255Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947346347Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947427868Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94751368Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947659922Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947683463Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947743554Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947773295Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947839166Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947866746Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947929418Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.947957938Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94805729Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948145062Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948179572Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948228613Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948285245Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948320405Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948373646Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948401257Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948460308Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948469838Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94854021Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948692782Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.948873546Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.949010388Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.949212092Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.949411336Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.949418596Z namespaces/openshift-multus/pods/multus-6qcwf/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.949453217Z namespaces/openshift-multus/pods/multus-6qcwf/multus-6qcwf.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94958289Z namespaces/openshift-multus/pods/multus-6qcwf/kube-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94959337Z namespaces/openshift-multus/pods/multus-6qcwf/kube-multus/kube-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.94959742Z namespaces/openshift-multus/pods/multus-6qcwf/kube-multus/kube-multus/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.949635411Z namespaces/openshift-multus/pods/multus-6qcwf/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950315224Z namespaces/openshift-multus/pods/multus-6qcwf/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950386195Z namespaces/openshift-multus/pods/multus-6qcwf/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950423476Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950477977Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/multus-additional-cni-plugins-2z6v6.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9506472Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/bond-cni-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95066314Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950671591Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950684211Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950777302Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950850324Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950867544Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950875084Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/cni-plugins/cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950927165Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.950975696Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951060238Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951127499Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95117659Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/egress-router-binary-copy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95118533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95119019Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951201391Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951302773Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951511717Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951550607Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951559868Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951564508Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951569268Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951573888Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951616629Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951658539Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/routeoverride-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95167008Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95167499Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95170011Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951791122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951860053Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951903354Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni-bincopy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951912084Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951916854Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.951935775Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952023977Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952107538Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952138669Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952146969Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952150939Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95218898Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952272881Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952345093Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2z6v6/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952373873Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952417164Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/multus-additional-cni-plugins-qgsxj.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952560427Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/bond-cni-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952571507Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952575537Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952610038Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952699939Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952765861Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952796521Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952806312Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/cni-plugins/cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952811521Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952845692Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.952937194Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953004765Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953033556Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/egress-router-binary-copy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953040776Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953044806Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953078777Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953169108Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95323675Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95326218Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95327233Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953282121Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953313271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953375843Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953452444Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953479805Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/routeoverride-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953484774Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953488715Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953540796Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953632457Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953703889Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953734719Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni-bincopy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95373955Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95374577Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95378989Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953877952Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953951463Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953983804Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953990264Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.953994304Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954030855Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954119987Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954186398Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qgsxj/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95427272Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95431849Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/multus-additional-cni-plugins-t4m8f.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954445403Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/bond-cni-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954453533Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954457483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954488124Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954605996Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954670987Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954701958Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954707738Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/cni-plugins/cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954712038Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954753159Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954844851Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954911852Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954937733Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/egress-router-binary-copy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954942073Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954948043Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.954989223Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955080105Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955148417Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955174597Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955180767Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955184947Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955225108Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955298939Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955365841Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955391321Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/routeoverride-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955395771Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955401651Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955447032Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955540024Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955624436Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955650166Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni-bincopy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955655546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955661836Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955701937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955791469Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95586239Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955893891Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955908261Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955915051Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.955938342Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956034134Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956103315Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-t4m8f/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956130876Z namespaces/openshift-multus/pods/multus-jv2d4/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956163676Z namespaces/openshift-multus/pods/multus-jv2d4/multus-jv2d4.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956260718Z namespaces/openshift-multus/pods/multus-jv2d4/kube-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956267778Z namespaces/openshift-multus/pods/multus-jv2d4/kube-multus/kube-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956271788Z namespaces/openshift-multus/pods/multus-jv2d4/kube-multus/kube-multus/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.956298299Z namespaces/openshift-multus/pods/multus-jv2d4/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957108604Z namespaces/openshift-multus/pods/multus-jv2d4/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957178265Z namespaces/openshift-multus/pods/multus-jv2d4/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957199636Z namespaces/openshift-multus/pods/multus-ns9fg/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957251397Z namespaces/openshift-multus/pods/multus-ns9fg/multus-ns9fg.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957342669Z namespaces/openshift-multus/pods/multus-ns9fg/kube-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957355709Z namespaces/openshift-multus/pods/multus-ns9fg/kube-multus/kube-multus/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957360989Z namespaces/openshift-multus/pods/multus-ns9fg/kube-multus/kube-multus/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.957386369Z namespaces/openshift-multus/pods/multus-ns9fg/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958248806Z namespaces/openshift-multus/pods/multus-ns9fg/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958319438Z namespaces/openshift-multus/pods/multus-ns9fg/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958345758Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958377969Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon-ggjz7.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958481741Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958488811Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958492671Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958539252Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958652144Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958752056Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958790407Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958797587Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958801517Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.958826577Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95899466Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959063262Z namespaces/openshift-multus/pods/network-metrics-daemon-ggjz7/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959100153Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959172574Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon-gwxtd.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959253835Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959265896Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959270026Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959280496Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959380658Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959446869Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95950093Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95950842Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95951252Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959571262Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959738425Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959804556Z namespaces/openshift-multus/pods/network-metrics-daemon-gwxtd/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959839607Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.959913408Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon-pfxts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.95999677Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96000472Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96000883Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96003898Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960144022Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960210904Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960245604Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960254565Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960273455Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960288185Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960461869Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9605415Z namespaces/openshift-multus/pods/network-metrics-daemon-pfxts/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960568681Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960624322Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960669353Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960677403Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960723954Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960809585Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960853706Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960887147Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.960979289Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961014899Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96104631Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961110401Z namespaces/openshift-must-gather-xlxbw/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961118691Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961122801Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961148662Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961356696Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96158068Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961721673Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.961919587Z namespaces/openshift-must-gather-xlxbw/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962220022Z namespaces/openshift-network-console/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962262973Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962319444Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962351445Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962407106Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962433066Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962543309Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962668051Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962764043Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962810954Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962842104Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962920896Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962967997Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.962990747Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963048468Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96312908Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963180631Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963196641Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963293583Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963338834Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963384795Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963498217Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963589529Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963711941Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963782712Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.963925085Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964062318Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96419096Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964246961Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964284032Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964340823Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964383214Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964422585Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964459945Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964504086Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96466874Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96469524Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964747941Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964836413Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964911354Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.964961065Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965002826Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965048777Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965065327Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965128498Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965137198Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965165909Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965324342Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965501945Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965660928Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.965857972Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966054456Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966061896Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966100827Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin-6cf856758c-w5vzg.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966179138Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966186199Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966190229Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966221599Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966314881Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966384662Z namespaces/openshift-network-console/pods/networking-console-plugin-6cf856758c-w5vzg/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966428033Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966443394Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966504795Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966565026Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966610667Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966657388Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966712589Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966740179Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9667968Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966840611Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.966957713Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967055895Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967153527Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967182538Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967237049Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96730327Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967344181Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967400472Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967437163Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967488214Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967531015Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967630826Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967673027Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967720168Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96779826Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967848911Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967885571Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.967978573Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.968083615Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.968270019Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96834519Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.968661226Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.968747828Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96885785Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.968935211Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.968971302Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969026383Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969059074Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969149926Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969197747Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969244157Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969316049Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96935649Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96939458Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969443041Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969536913Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969646275Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969691906Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969718406Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969781638Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969826848Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969858949Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.96991227Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969959961Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969968001Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.969998222Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970151475Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970330478Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970466421Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970687955Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970873869Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970881949Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.970903179Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/network-check-source-869dcfcdcd-kns7z.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971002121Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/check-endpoints/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971009851Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/check-endpoints/check-endpoints/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971013882Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971045522Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971155794Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971223006Z namespaces/openshift-network-diagnostics/pods/network-check-source-869dcfcdcd-kns7z/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971262196Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971280607Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-bhf5x.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971377098Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-container/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971387949Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971394119Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971401749Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971497541Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971584662Z namespaces/openshift-network-diagnostics/pods/network-check-target-bhf5x/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971610243Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971656394Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-dqvff.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971734025Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-container/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971741095Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971751285Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971799557Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.971884568Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97195162Z namespaces/openshift-network-diagnostics/pods/network-check-target-dqvff/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97197793Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972026231Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-whhxk.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972111863Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-container/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972119383Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972123323Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972160693Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972250165Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972316267Z namespaces/openshift-network-diagnostics/pods/network-check-target-whhxk/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972354807Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972379968Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972434899Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972442999Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97248075Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972661503Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972703474Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972751755Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972833946Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972890447Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972907698Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.972973119Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97302106Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973081581Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973115842Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973165033Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973192153Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973284505Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973362267Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973453868Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973499419Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97355739Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973597401Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973646182Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973726664Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973776604Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973815475Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973897897Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973944028Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973952578Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.973984479Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97404282Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97408401Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974177232Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974260204Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974342635Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974424037Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974506878Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974667982Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974775494Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974826525Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974869365Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974922686Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.974966067Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975009768Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975055489Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97508457Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975134351Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975216962Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975292204Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975345375Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975383515Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975431716Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975463847Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975532178Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975545489Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975567539Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975724602Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.975913916Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976200571Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976384715Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976600109Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976625009Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9766756Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9766848Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976722111Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976773792Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976801133Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976862054Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976894195Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976943615Z namespaces/openshift-network-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.976990086Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977047167Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977090488Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977137259Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97717154Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977281532Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977361593Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977448855Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977489626Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977547757Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977617838Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977650629Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97770709Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977728Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977808412Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977862413Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977893434Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.977979155Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978013116Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978017576Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978068147Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978128388Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978159859Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978302831Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978383003Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978553246Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978641278Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978817132Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.978900543Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979022595Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979079637Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979119287Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979203419Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97926324Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979308291Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979356872Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979409543Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979459324Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979501694Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979566996Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979588626Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979684088Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979759119Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.97980804Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979842371Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979898292Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979940883Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.979996184Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980036195Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980115476Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980130737Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980185538Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980195578Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980232889Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980382691Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980638187Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980789999Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.980989923Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981174817Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981182307Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981213227Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter-b7cc9.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981294369Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981301509Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981310019Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9813408Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981428871Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981491573Z namespaces/openshift-network-operator/pods/iptables-alerter-b7cc9/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981533473Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981574624Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter-m8wzb.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981746148Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981753478Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981757488Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981793349Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98187986Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981948721Z namespaces/openshift-network-operator/pods/iptables-alerter-m8wzb/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.981985822Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982025483Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter-tdqd5.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982099835Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982106875Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982116025Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982160506Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982250727Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982317919Z namespaces/openshift-network-operator/pods/iptables-alerter-tdqd5/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.9823648Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98237634Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982434961Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982478202Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982535333Z namespaces/openshift-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982546333Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982550423Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982590864Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982757497Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.982969811Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983110534Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983305348Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983493151Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983541242Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983623284Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983665575Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983707946Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983752786Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983835718Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98391633Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.983990491Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984044812Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984090133Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984139484Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984185035Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984262906Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984306947Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984346078Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984429699Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98447414Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984531751Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984634673Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984722005Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984817777Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.984932219Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985009551Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985126623Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985321046Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985391168Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985421198Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98547748Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98553113Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985576751Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985624062Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985717704Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985791646Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985846547Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985885157Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985934008Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.985980499Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98604493Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986062281Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986147862Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986155233Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986184573Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986334706Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986512859Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986675552Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.986865896Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98705225Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98708106Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987142992Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987174922Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987345175Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987375886Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987425757Z namespaces/openshift-operators/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987433177Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987437227Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987470178Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987652201Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987828285Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.987963387Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988176631Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988362795Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988390435Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988452266Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988496757Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98864492Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988688611Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988742472Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988775553Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.988971867Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989049258Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989127499Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98917786Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989208431Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989344104Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989380304Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989423265Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989463666Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989626199Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.98966717Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989710491Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989794482Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989844523Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989852793Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989879974Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989932415Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.989975026Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99017047Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.990254251Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.990603428Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.990742471Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991286351Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991371673Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991539776Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991613327Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991646118Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99174172Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991857372Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991888293Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.991981614Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992030655Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992065696Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992132817Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992166348Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992236099Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99226934Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992320921Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992329341Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992363322Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992422293Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992450694Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992557536Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992642157Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992678718Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992712499Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99277358Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99280168Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992889342Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992917832Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992978693Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.992985824Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993019004Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993110136Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993156607Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993201868Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993249979Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993258339Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993286139Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993433042Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993632956Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993772349Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.993960583Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994155016Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994163486Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994215287Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-node-jwnj6.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994384261Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994391491Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994395481Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994465542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994602975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994669036Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994709297Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994716427Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994720377Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994755088Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99487777Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994945511Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994972912Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/nbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994977162Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/nbdb/nbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.994980902Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/nbdb/nbdb/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995026153Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/nbdb/nbdb/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995124165Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995185986Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995219547Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/northd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995224297Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/northd/northd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995228177Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/northd/northd/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995266678Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/northd/northd/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99537324Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995440621Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/northd/northd/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995478932Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-acl-logging/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995484862Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995489092Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995531693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995632525Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995686296Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995693876Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-controller/ovn-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995697836Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.995730277Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.996017242Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.996105204Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.996122804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.996128414Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.996135434Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.996175635Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.997895838Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99796786Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998065871Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/sbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998078522Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/sbdb/sbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998097522Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/sbdb/sbdb/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998113602Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/sbdb/sbdb/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998216664Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998266725Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-jwnj6/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998310346Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998379597Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-node-sc5lg.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.99853772Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998550251Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998558681Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998608892Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998737644Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998806145Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998849906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998857946Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998862847Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.998870337Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999002339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999067281Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999090141Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/nbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999098301Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/nbdb/nbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999102401Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/nbdb/nbdb/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999148142Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/nbdb/nbdb/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999245274Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999313155Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999333316Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/northd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999339856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/northd/northd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999348766Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/northd/northd/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999391637Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/northd/northd/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999510119Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999604661Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/northd/northd/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999641092Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-acl-logging/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999648842Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999652852Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999681822Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999772434Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999827725Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999835716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-controller/ovn-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999839636Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:54.999860976Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.000170002Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.000234173Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.000271814Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.000279174Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.000283004Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.000309165Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002015027Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002091779Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002109019Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/sbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002116709Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/sbdb/sbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002126969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/sbdb/sbdb/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00216806Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/sbdb/sbdb/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002267372Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002344144Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sc5lg/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002364934Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002436906Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-node-sh5ck.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002620829Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002633769Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002638949Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00266506Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002775042Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002842473Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002869474Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002876214Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002880184Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.002919815Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003029787Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003131029Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.0031744Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/nbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00318693Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/nbdb/nbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00319183Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/nbdb/nbdb/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00319959Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/nbdb/nbdb/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003309542Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003378553Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003393404Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/northd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003402884Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/northd/northd/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003408684Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/northd/northd/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003460715Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/northd/northd/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003603528Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003672309Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/northd/northd/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00369673Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-acl-logging/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00370162Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00370945Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003751671Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003841492Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003883063Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003889803Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-controller/ovn-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003895233Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.003937104Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00424438Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.004308461Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.004336822Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.004343222Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.004347132Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.004384343Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006161707Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006229348Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006262659Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/sbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006268109Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/sbdb/sbdb/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006272789Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/sbdb/sbdb/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00630954Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/sbdb/sbdb/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006409532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006476383Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sh5ck/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006537324Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006573485Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006630606Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006639546Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006664947Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006761689Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006883211Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006931932Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.006973393Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007056664Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007137936Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007188797Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007254618Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007310699Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007319519Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007323589Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.00735048Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007403481Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007410551Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007449032Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.007683616Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.0078981Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008034663Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008249127Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008534112Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008606594Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008663915Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008704136Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008758637Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008796928Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008879009Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.008984671Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009083773Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009129954Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009159684Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009210285Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009253866Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009333968Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009378819Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009408539Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009497651Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009555122Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009565172Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009608893Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009659004Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009695955Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009815767Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.009895079Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010089533Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010166944Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010296686Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010419349Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010631533Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010685564Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010728525Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010781186Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010817507Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010868317Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.010911738Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01099015Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011070921Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011109782Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011157173Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011212224Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011249205Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011291305Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011337637Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011391427Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011401798Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011412848Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011598762Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011778775Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.011921288Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012133322Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012318025Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012326776Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012351046Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/service-ca-operator-7695fc6c8c-n7xwx.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012434087Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/service-ca-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012441538Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012445728Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012475838Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012679302Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-7695fc6c8c-n7xwx/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012833985Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012865986Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012913667Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012944937Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.012999508Z namespaces/openshift-service-ca/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013031909Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.0131069Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013142031Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013194162Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013220643Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013312884Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013411356Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013509758Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013570129Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01362334Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013665131Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013700462Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013784733Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013816734Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013868715Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013947977Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013980507Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.013987007Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014031068Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014086029Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01412608Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014303023Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014379865Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014493067Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014590019Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014699601Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.014821753Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015014387Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015064088Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015092769Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01514979Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01518139Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015234711Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015260652Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015349623Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015429145Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015464806Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015495956Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015574538Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015614159Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015663909Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01567285Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01569539Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.015845833Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016025487Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016174859Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016363203Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016578007Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016592967Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016620658Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-79f68b9f4f-npn7x.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01670476Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01671714Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01672329Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01674663Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.016956134Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017018226Z namespaces/openshift-service-ca/pods/service-ca-79f68b9f4f-npn7x/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017050076Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017097947Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017144658Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017181419Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01723068Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017278701Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017337992Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017372822Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017422974Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017465054Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017566076Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017646938Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017733709Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01774941Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017811751Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017855162Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017881272Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.017976174Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018000534Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018040015Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018128797Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018158237Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018199508Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01830608Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018379542Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018460523Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018557285Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.018708208Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01882803Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019005814Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019062085Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019103486Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019149717Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019182547Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019226218Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019263359Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019350431Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019427332Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019462443Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019496093Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019581145Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019599515Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019651896Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019661397Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.019686487Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.01985139Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020032613Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020175586Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020446381Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020686696Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020720447Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020779308Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020822649Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02086756Z namespaces/openshift/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020915161Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.020972122Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021004672Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021054343Z namespaces/openshift/apps/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021083084Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021195866Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021271377Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021355769Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02142789Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021462341Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021533382Z namespaces/openshift/batch/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021570823Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021665045Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021711496Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021724546Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021827088Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021853689Z namespaces/openshift/core/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.021904549Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022030682Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022132094Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022231946Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022322808Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02243287Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022612613Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022810797Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022869058Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.022917629Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02298253Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.023014291Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024408548Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024455959Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024567301Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024655793Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024763195Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024871897Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.024955938Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.025118561Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.025287105Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.025456578Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02558954Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.025702293Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.025839525Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.025961858Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02609242Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.026200062Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.026308124Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.026430996Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02658111Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.026726072Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.026835494Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.026946606Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027070929Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027185661Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027305953Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027419636Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027536668Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02766565Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027792843Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.027918325Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.028040547Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02818338Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.028310443Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.028415605Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.028638879Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.028773392Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.028896774Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029055067Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029142289Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029265351Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029364283Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029485345Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029648418Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.02973736Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029827262Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.029958534Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.030136148Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.03024841Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.030359012Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.030437864Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.030529645Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.030652738Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.0307715Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.030893712Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031014465Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031126647Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031243519Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031335861Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031433323Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031583806Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031670357Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031716428Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.03180074Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031879651Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031923482Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.031976243Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032018964Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032054265Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032121336Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032131416Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032161197Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032384081Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032587035Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032726428Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.032929292Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033121225Z namespaces/openshift/policy/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033150286Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033205367Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033245558Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033276778Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033284638Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033324499Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033441851Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033590864Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033701586Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033809318Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.033976281Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034082784Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034200696Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034312248Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.03443327Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034566993Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034687945Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034810938Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.034973081Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035074853Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035203215Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035306737Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035401979Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035540882Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035665794Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035770716Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.035872218Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.03597488Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036100962Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036199484Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036305086Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036404128Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.03650514Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036658553Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036772365Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036873927Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.036979759Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037077481Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037206664Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037384097Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037483609Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037616812Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037716633Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037834096Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.037981469Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.038088091Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.038208093Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.038327715Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.038452448Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.03860391Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.038731733Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.038882946Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039012768Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039143431Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039260763Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039346575Z network_logs/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039370325Z network_logs/cluster_scale [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039463767Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039540479Z network_logs/multi-networkpolicy [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039646621Z network_logs/net-attach-def [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039725462Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039875735Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.039966927Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.042405223Z nodes/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.042432154Z nodes/debug [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.042538246Z nodes/ip-10-0-132-16.ec2.internal/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.042604237Z nodes/ip-10-0-132-16.ec2.internal/cpu_affinities.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.042937384Z nodes/ip-10-0-132-16.ec2.internal/dmesg [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.043228469Z nodes/ip-10-0-132-16.ec2.internal/ethtool_channels [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.043303031Z nodes/ip-10-0-132-16.ec2.internal/ethtool_features [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.043393173Z nodes/ip-10-0-132-16.ec2.internal/ip-10-0-132-16.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.043898062Z nodes/ip-10-0-132-16.ec2.internal/irq_affinities.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.044061395Z nodes/ip-10-0-132-16.ec2.internal/lscpu [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.044192158Z nodes/ip-10-0-132-16.ec2.internal/lspci [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.04428506Z nodes/ip-10-0-132-16.ec2.internal/podresources.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.044394272Z nodes/ip-10-0-132-16.ec2.internal/pods_info.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.044538385Z nodes/ip-10-0-132-16.ec2.internal/proc_cmdline [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.044652067Z nodes/ip-10-0-132-16.ec2.internal/sysinfo.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.046564174Z nodes/ip-10-0-132-16.ec2.internal/sysinfo.tgz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.046717986Z nodes/ip-10-0-141-118.ec2.internal/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.046754037Z nodes/ip-10-0-141-118.ec2.internal/cpu_affinities.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.047061833Z nodes/ip-10-0-141-118.ec2.internal/dmesg [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.047325368Z nodes/ip-10-0-141-118.ec2.internal/ethtool_channels [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.04740603Z nodes/ip-10-0-141-118.ec2.internal/ethtool_features [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.047486241Z nodes/ip-10-0-141-118.ec2.internal/ip-10-0-141-118.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.04793296Z nodes/ip-10-0-141-118.ec2.internal/irq_affinities.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.048095763Z nodes/ip-10-0-141-118.ec2.internal/lscpu [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.048220705Z nodes/ip-10-0-141-118.ec2.internal/lspci [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.048307227Z nodes/ip-10-0-141-118.ec2.internal/podresources.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.048416569Z nodes/ip-10-0-141-118.ec2.internal/pods_info.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.048540032Z nodes/ip-10-0-141-118.ec2.internal/proc_cmdline [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.048641894Z nodes/ip-10-0-141-118.ec2.internal/sysinfo.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.050470138Z nodes/ip-10-0-141-118.ec2.internal/sysinfo.tgz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.050633082Z nodes/ip-10-0-141-44.ec2.internal/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.050667802Z nodes/ip-10-0-141-44.ec2.internal/cpu_affinities.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.050957028Z nodes/ip-10-0-141-44.ec2.internal/dmesg [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.051209333Z nodes/ip-10-0-141-44.ec2.internal/ethtool_channels [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.051287324Z nodes/ip-10-0-141-44.ec2.internal/ethtool_features [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.051371906Z nodes/ip-10-0-141-44.ec2.internal/ip-10-0-141-44.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.051830555Z nodes/ip-10-0-141-44.ec2.internal/irq_affinities.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.051985798Z nodes/ip-10-0-141-44.ec2.internal/lscpu [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.05212637Z nodes/ip-10-0-141-44.ec2.internal/lspci [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.052216712Z nodes/ip-10-0-141-44.ec2.internal/podresources.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.052319664Z nodes/ip-10-0-141-44.ec2.internal/pods_info.json [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.052429806Z nodes/ip-10-0-141-44.ec2.internal/proc_cmdline [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.052511958Z nodes/ip-10-0-141-44.ec2.internal/sysinfo.log [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.054435065Z nodes/ip-10-0-141-44.ec2.internal/sysinfo.tgz [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.054639869Z pod_network_connectivity_check/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.054662929Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.054763741Z static-pods/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.054772401Z static-pods/kube-apiserver/ [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.062754945Z [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.062785435Z sent 71,368 bytes received 10,307,700 bytes 6,919,378.67 bytes/sec [must-gather] [must-gather-5lnvf] OUT 2026-07-09T18:41:55.062791755Z total size is 123,193,561 speedup is 11.87 [must-gather] [must-gather ] OUT 2026-07-09T18:41:55.362498626Z namespace/openshift-must-gather-xlxbw deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: f73a085b-0234-456f-8f79-621ceb1f1251 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.24" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-mgt82 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-mgt82-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-mgt82 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 18537394 Jul 9 18:43 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-mgt82/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 60e0e36] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-mgt82/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] fb3aecb..60e0e36 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-09T18:43:49.477Z","type":3}]