task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-ht9xc-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-ht9xc-kubeconfig [get-kubeconfig] Found admin password secret: cluster-ht9xc-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-ht9xc-password [get-kubeconfig] API Server URL: https://a943d576bb3374d21a46773ce0e41ed6-04a49dcb808c47e0.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.554d0026-27d2-42dc-9d76-4c718de147a7.prod.konfluxeaas.com [clone-repo] chore-autogluon-midstream-dockerfile [clone-repo] https://github.com/DorotaDR/kserve [clone-repo] Cloning into '/workspace/source'... [clone-repo] Updating files: 93% (2901/3108) Updating files: 94% (2922/3108) Updating files: 95% (2953/3108) Updating files: 96% (2984/3108) Updating files: 97% (3015/3108) Updating files: 98% (3046/3108) Updating files: 99% (3077/3108) Updating files: 100% (3108/3108) Updating files: 100% (3108/3108), done. [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:b91ab915596b4f349832befac95e931936365ada90787fcb1c8ef485ef1e3a3a [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:b91ab915596b4f349832befac95e931936365ada90787fcb1c8ef485ef1e3a3a [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:5226294b4773f2e497f4c6afcdca0755c00c8cb5c867d6a7bdbc722775b68ea5 [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:5226294b4773f2e497f4c6afcdca0755c00c8cb5c867d6a7bdbc722775b68ea5 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:264e4696bc4acb7c1e00cabab43621f26f8fb70223f71f60afed1f1b1add26b9 [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:264e4696bc4acb7c1e00cabab43621f26f8fb70223f71f60afed1f1b1add26b9 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:673b6a60d37e82f128f4a0fe4fe7213c3a49392b21477d491e45a2e71a8ceb39 [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:673b6a60d37e82f128f4a0fe4fe7213c3a49392b21477d491e45a2e71a8ceb39 [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] INFERENCE_POOL_GROUP=inference.networking.k8s.io (detected from OCP 4.21.23) [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1684 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1684 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1684 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Prepared 101 packages in 1.80s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 416ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 46ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a943d576bb3374d21a46773ce0e41ed6-04a49dcb808c47e0.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-wvsk4 1/1 Running 0 42s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-wvsk4 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-trgf8 1/1 Running 0 47s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-trgf8 condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-nzv4p 1/1 Running 0 52s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-nzv4p condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:5226294b4773f2e497f4c6afcdca0755c00c8cb5c867d6a7bdbc722775b68ea5 [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:673b6a60d37e82f128f4a0fe4fe7213c3a49392b21477d491e45a2e71a8ceb39 [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:b91ab915596b4f349832befac95e931936365ada90787fcb1c8ef485ef1e3a3a [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:264e4696bc4acb7c1e00cabab43621f26f8fb70223f71f60afed1f1b1add26b9 [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-multi-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-pd-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-single-node-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-57d57b58b4-5rf4k 0/1 ContainerCreating 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-57d57b58b4-5rf4k condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.23) [e2e-llm-inference-service] 🎯 Server version (4.21.23) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.20.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 found, but not yet Succeeded (Phase: Installing). Waiting... (10/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-559pb 1/1 Running 0 11s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-559pb condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-m2d5h 0/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-m2d5h condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-655f7c58b4-962b9 1/1 Running 0 7s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-655f7c58b4-962b9 condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-xhvv8 1/1 Running 0 65s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-xhvv8 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-xhvv8 1/1 Running 0 76s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-xhvv8 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-5876b4f96d-hvhnk" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-5876b4f96d-pgx27 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/autogluon-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.23 [e2e-llm-inference-service] Kubernetes Version: v1.34.8 [e2e-llm-inference-service] ClusterVersion desired: 4.21.23 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.23 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.20.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:5226294b4773f2e497f4c6afcdca0755c00c8cb5c867d6a7bdbc722775b68ea5 [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:4cd1e65fa018823f7402c14c1513e0eefc07187497791c87132f3f45673e984d [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:5d13ab3c20c8da1064020d19c1d95fd5a563c47cac7c19e4051ee53b992a80da [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:673b6a60d37e82f128f4a0fe4fe7213c3a49392b21477d491e45a2e71a8ceb39 [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:09797be92ca71fe79970c58d8f35eb172fe5a0953f4f552e41be30414526f7da [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-570.122.1.el9_6.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'anyio': '4.9.0', 'httpx': '0.30.0', 'metadata': '3.1.1', 'xdist': '3.6.1', 'cov': '5.0.0', 'asyncio': '0.23.8', 'json-report': '1.5.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: anyio-4.9.0, httpx-0.30.0, metadata-3.1.1, xdist-3.6.1, cov-5.0.0, asyncio-0.23.8, json-report-1.5.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-09 09:46:17.514 6141 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-09 09:46:17.514 6138 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-09 09:46:17.578 6138 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 09:46:17.578 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 09:46:17.601 6138 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 09:46:17.601 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 09:46:17.609 6138 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 09:46:17.609 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-09 09:46:34.815 6138 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 09:46:34.815 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 09:46:34.861 6138 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 09:46:34.861 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-09 09:56:30.319 6138 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 09:56:30.319 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 09:56:30.369 6138 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 09:56:30.369 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-09 10:08:55.552 6138 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:08:55.552 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:08:55.604 6138 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 10:08:55.604 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-09 10:08:55.604 6138 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:08:55.604 6138 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:08:55.608 6138 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 10:08:55.608 6138 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 10:08:55.621 6138 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 10:08:55.621 6138 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-09 10:08:55.621 6138 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:08:55.621 6138 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:08:55.625 6138 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 10:08:55.625 6138 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 10:08:55.634 6138 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-09 10:08:55.634 6138 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-09 10:24:07.309 6138 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:24:07.309 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:24:07.347 6138 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 10:24:07.347 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 10:24:07.355 6138 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 10:24:07.355 6138 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-09 10:24:07.356 6138 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:24:07.356 6138 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:24:07.360 6138 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 10:24:07.360 6138 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 10:24:07.372 6138 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 10:24:07.372 6138 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-09 10:24:07.373 6138 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:24:07.373 6138 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-09 10:24:07.376 6138 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 10:24:07.376 6138 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 10:24:07.387 6138 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-09 10:24:07.387 6138 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:816: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-v0... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': None}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-09T10:40:08.417532', start_time = 1783593608.4179103 [e2e-llm-inference-service] duration = 1103.0843234062195, timestamp_end = '2026-07-09T10:58:31.502236' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7f06d460a660> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 502: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1586 Checking LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1612 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1622 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-09T10:39:07.501536] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1637 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-09T10:39:07.513875] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-09T10:39:07.726115] end - ✅ in 0.212s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-09T10:39:07.726249] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-09T10:39:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'message': 'Inference Pool kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-09T10:39:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'WorkloadsReady', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-09T10:39:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'Ready', 'RouterReady'}, expected {'Ready', 'WorkloadsReady', 'RouterReady'}, got [{'lastTransitionTime': '2026-07-09T10:39:39Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-09T10:39:42Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:41Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:32Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-09T10:39:39Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-09T10:39:42Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-09T10:40:08.417364] end - ✅ in 60.691s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-09T10:40:08.417532] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': None} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-09T10:40:08.417915] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-09T10:40:08.432605] end - ✅ in 0.014s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://aa55e646e69d84880aa52a79336c5b53-1395608746.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Service returned 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-09T10:46:17.799139] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-09T10:46:17.869868] end - ✅ in 0.070s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://aa55e646e69d84880aa52a79336c5b53-1395608746.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-09T10:52:27.168664] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-09T10:52:27.211478] end - ✅ in 0.042s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://aa55e646e69d84880aa52a79336c5b53-1395608746.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-09T10:58:31.502236] end - ❌ 1103.084s: Service returned 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] ❌ ERROR: Failed to call llm inference service scheduler-v06-pd-migration-test: Service returned 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'scheduler-v06-pd-migration-test' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService scheduler-v06-pd-migration-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:07Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:07Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:23Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:40:07Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '71081' [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] - name: scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] - name: workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://aa55e646e69d84880aa52a79336c5b53-1395608746.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://aa55e646e69d84880aa52a79336c5b53-1395608746.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:42Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:32Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:40:07Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:40:07Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:40:07Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:42Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://aa55e646e69d84880aa52a79336c5b53-1395608746.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.33:8000/health": dial tcp 10.133.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:10 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-56f8b9c98b-rkwkv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-866757c5f5-xgvsh to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:05 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:05 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:05 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:06 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:06 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:06 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:10 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-866757c5f5-xgvsh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-866757c5f5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-56f8b9c98b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:51 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:18 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 26.94s (26.94s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:18 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:18 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:07 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.26:8000/health": dial tcp 10.133.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.26:8000/health": dial tcp 10.133.0.26:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-667d764b96-zv8h2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:49 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.482s (1.482s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:49 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:49 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-944f8664b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-667d764b96 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:17 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:24 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:24 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:24 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:10 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.29:8000/health": dial tcp 10.133.0.29:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.29:8000/health": dial tcp 10.133.0.29:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-97b67b5bd-7fwhr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:21 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-555b6fb47b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-97b67b5bd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:34 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-criticality": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-criticality": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:34 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-lora-crit": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-lora-crit": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:34 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-round-trip": llminferenceservices.serving.kserve.io "conv-test-round-trip" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:33 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-v1a1-to-v1a2": llminferenceservices.serving.kserve.io "conv-test-v1a1-to-v1a2" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.48:8001/health": dial tcp 10.133.0.48:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:23:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:23:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:18 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.49:8000/health": dial tcp 10.133.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:23:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-76797blfdh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-76797blfdh to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:16 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:16 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:16 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:23:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:23:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-76797bb4cb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-f8f6599f9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:23:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:36 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:36 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:36 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:52 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:35 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:51 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.38:8000/health": dial tcp 10.133.0.38:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-5cf8847bb6-mgmhx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-54c5987dcxrmw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54c5987dcxrmw to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:34 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:34 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:34 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:35 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:35 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-54c5987dbf from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-5cf8847bb6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:16 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:02 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-xvtw2 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:40 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:44 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:44 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:44 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:35 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-xvtw2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:53 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:35 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:35 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:35 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-58b4cc556f-c2msx_d515b963-ccfe-45eb-9e48-361ea7fc2fa5 Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:37 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-58b4cc556f-c2msx_d515b963-ccfe-45eb-9e48-361ea7fc2fa5 Normal ProvisioningSucceeded Successfully provisioned volume pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.24/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.111s (1.111s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:34 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-2f0a622e": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-2f0a622e": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bg6222 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bg6222 to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:56 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.085s (1.085s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:56 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:56 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:34 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:49 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.32:8000/health": dial tcp 10.134.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:56 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "gw-section-name-router-with-gat-f1d92d0f": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "gw-section-name-router-with-gat-f1d92d0f": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.55:8000/health": dial tcp 10.133.0.55:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:23 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:39 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.55:8000/health": dial tcp 10.133.0.55:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:02 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:15 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:23 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.45:8001/health": dial tcp 10.133.0.45:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:23 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.46:8000/health": dial tcp 10.133.0.46:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c9d4dfcd7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:42 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:34 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:45 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:45 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:45 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:31 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.36:8000/health": dial tcp 10.133.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:45 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:01 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.36:8000/health": dial tcp 10.133.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:41 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:33:13 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.57:8000/health": dial tcp 10.133.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:45 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.57:8000/health": dial tcp 10.133.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 848554f896 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:33:24 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:25 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4rg26k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4rg26k to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:44 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:44 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:44 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:44 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:13 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:41 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:56 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:53 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.34:8000/health": dial tcp 10.133.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:27 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.34:8000/health": dial tcp 10.133.0.34:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schemzdr5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-6456d6d6c6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schemzdr5 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:27 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:27 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:39 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:03 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:57 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:57 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:57 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:59:03 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.35:8000/health": dial tcp 10.133.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.35:8000/health": dial tcp 10.133.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schefrzdw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-6b99777769 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schefrzdw to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:55 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:55 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:55 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:14 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:14 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:59:13 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:39 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:57 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.41:8000/health": dial tcp 10.133.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:59 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:15 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.41:8000/health": dial tcp 10.133.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:10:07 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:29 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-2577e794] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:11 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.37:8000/health": dial tcp 10.133.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:03 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.37:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:39 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:13 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 1.219s (1.219s including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:14 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:42 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.39:8001/health": dial tcp 10.133.0.39:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:13 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:13 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:13 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:42 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.40:8000/health": dial tcp 10.133.0.40:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:10 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:09 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvkpb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4qvkpb to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:54 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:53 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:44 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:59 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:29 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:23 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-jbwx5 to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:08 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:23 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.36:8000/health": dial tcp 10.134.0.36:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-jbwx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:12 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4lxfbj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4lxfbj to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:15 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:31 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.35:8000/health": dial tcp 10.134.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:42 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:18 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xtk2x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879xtk2x to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:11 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:53:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:54:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:37 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:39 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:59 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:59 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:07:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7mkkm to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.70/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:55 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7mkkm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-5bb8bf44b6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegqkhs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegqkhs to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.74/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:20 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:54 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schev5dqt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-7d998647d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schev5dqt to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:48 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:16:48 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefq7b7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefq7b7 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:38 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:38 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:38 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:38 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheqg54s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheqg54s to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:01:11 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:03:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecdx9s to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:00:45 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schecdx9s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:58:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-5df964d74b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8n9dk to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:59 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:51:59 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche8n9dk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:50:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schewcdcn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schewcdcn to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2, llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:04:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:58 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:06:58 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesztcv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schesztcv to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:40 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:08 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:55:08 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehd28x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schehd28x to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:49:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:52:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:41 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:41 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:41 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:00 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.54:8000/health": dial tcp 10.133.0.54:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-multiple-adapters-test-kserve-6f544c69b-vjtx4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-multiple-adapters-test-kserve-6f544c69b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:26:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:11 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-multiple-adapters-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:20 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:20 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:20 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:22:26 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.51:8000/health": dial tcp 10.133.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:25:31 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.51:8000/health": dial tcp 10.133.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-single-adapter-test-kserve-64b48975c9-spbbv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-single-adapter-test-kserve-64b48975c9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:07 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:21:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:22:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-single-adapter-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-8sr5g to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" in 3.269s (3.269s including waiting). Image size: 302212544 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:56 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:57 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.232s (1.232s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:57 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:57 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:10 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:25 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.47:8000/health": dial tcp 10.132.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:51 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:51 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:51 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:10 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:26 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.25:8000/health": dial tcp 10.133.0.25:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-b49d646d8-c9pmj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-b49d646d8-8sr5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-78bbd8bb77thk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-78bbd8bb77thk to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" in 2.92s (2.92s including waiting). Image size: 302212544 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:50 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:52 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:53 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.303s (1.303s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:53 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 30.389s (30.389s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:38 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.48:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:10 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:10 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:10 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-78bbd8bb4b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-b49d646d8 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:42 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:09 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:48:10 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:03 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:03 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:03 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:23 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.60:8000/health": dial tcp 10.133.0.60:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:24 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:44 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:44 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:44 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:33:49 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.59:8000/health": dial tcp 10.133.0.59:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:03 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:19 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.59:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-789646768b-vjgdj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-789646768b-swqvb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-5496c57d7-2jm87 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.71/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:40 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:42 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:42 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:42 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:24 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-router-scheduler-5496c57d7-2jm87 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set prestop-hook-test-kserve-router-scheduler-5496c57d7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set prestop-hook-test-kserve-789646768b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/prestop-hook-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/prestop-hook-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/prestop-hook-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/prestop-hook-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/prestop-hook-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/prestop-hook-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/prestop-hook-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:00 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [prestop-hook-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:23 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [prestop-hook-test] is no longer Ready because of: MainWorkloadReady, WorkloadsReady [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:34:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:31 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.61:8000/health": dial tcp 10.133.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-54bc76f55b-7whj9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:16 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: rolling-upgrade-test-kserve-54bc76f55b-7whj9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-h8c88 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.73/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:06 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:06 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:06 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:22 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-859df57bcb-h8c88 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:33 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:22 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-54bc76f55b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:05 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-859df57bcb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:16 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set rolling-upgrade-test-kserve-54bc76f55b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:28 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy rolling-upgrade-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "rolling-upgrade-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/rolling-upgrade-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/rolling-upgrade-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/rolling-upgrade-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/rolling-upgrade-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:05 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [rolling-upgrade-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:36:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-857cf9f944-b8c7s to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:35 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:47 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:52 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.31:15021/healthz/ready": dial tcp 10.134.0.31:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:00 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-857cf9f944-rcfs8 to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:18 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:18 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:18 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:23 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-857cf9f944-wf4cz to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:30 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:30 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:30 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:32 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-857cf9f944-zw9hc to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:56 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:56 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:56 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:56 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:57 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.61:15021/healthz/ready": dial tcp 10.132.0.61:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:59 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-857cf9f944-rcfs8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:23 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-857cf9f944-rcfs8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-857cf9f944-b8c7s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:47:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-857cf9f944-b8c7s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-857cf9f944-wf4cz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-857cf9f944-wf4cz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:55 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-857cf9f944-zw9hc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-857cf9f944-zw9hc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-b49948d49-g6vgn to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:55 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:55 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:56:55 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:57:01 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.33:15021/healthz/ready": dial tcp 10.134.0.33:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-b49948d49-qcsxp to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:18 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:18 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:21 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 2.811s (2.812s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:21 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:21 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:38 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 503 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:17 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-b49948d49-qcsxp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:37 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-1-openshift-default-b49948d49-qcsxp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:57 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-b49948d49-g6vgn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:55 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-857cf9f944 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:17 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-b49948d49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:21 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:57 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-857cf9f944 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 09:46:37 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-1-openshift-default-b49948d49 from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:08:55 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-578d7b9c-m7zq9 to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:09 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:23 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:23 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:23 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:28 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.39:15021/healthz/ready": dial tcp 10.134.0.39:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-578d7b9c-m7zq9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-6f899b565b-q5dtd to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:08 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.132.0.67:15021/healthz/ready": dial tcp 10.132.0.67:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:12 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-6f899b565b-q5dtd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:11 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: router-gateway-2-openshift-default-6f899b565b-q5dtd [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-6f899b565b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-578d7b9c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:11 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set router-gateway-2-openshift-default-6f899b565b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:11 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:07 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.52:8001/health": dial tcp 10.133.0.52:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:29 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.53:8000/health": dial tcp 10.133.0.53:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:30 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-79f7f4c6d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-8bb68db4-fjqfq to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.68/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:26 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:26 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:26 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-8bb68db4-fjqfq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-8bb68db4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-5d7f47bbd7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:24:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:15 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:27:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:12 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:12 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:12 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:10:58 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.44:8000/health": dial tcp 10.133.0.44:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:16 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:32 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.44:8000/health": dial tcp 10.133.0.44:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-6b9f4b8d6b-97gl5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-64f4948457-zxj67 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:09 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:17 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:17 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-64f4948457-zxj67 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-64f4948457 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-6b9f4b8d6b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:09:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:11:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:13:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:33 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:48 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.64:8000/health": dial tcp 10.133.0.64:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-68f87zhjk7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87zhjk7 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.75/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:49 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:49 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:49 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:32 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:33 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:34 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:24 kserve-ci-e2e-test LLMInferenceServiceController Warning Error Reconciliation failed: failed to reconcile workload: failed to reconcile self-signed certificates secret: failed to reconcile self-signed TLS certificate: failed to create v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs: context canceled [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:37:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-configmap-ref-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-v5zvl to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:46 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:03 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.42:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-v5zvl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-567j8hmg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567j8hmg to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.76/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-567469f665 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:45 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:38:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dcgxfk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dcgxfk to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.77/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:43 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:43 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:03 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:40 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-s8n46 to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:51 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-s8n46 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-router-scheduler-d9ddfrx45x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfrx45x to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.72/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:51 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:51 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:11 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:50 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-inline-config-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:35:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.78/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 810ms (810ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn to ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test kubelet/ip-10-0-128-170.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.79/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:31 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-5c87756974 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:28 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:39:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-pd-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.68/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-prefill-5fc599dc49 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched8c47c to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.80/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-threshold-migration-test-kserve-router-sched8c47c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-router-scheduler-dbc65f645 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-threshold-migration-test-kserve-5994569d5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-threshold-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-threshold-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-threshold-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:40:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test0a07e82be2036793543b2d3db2ea0785-scheduler-v0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:39 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:39 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:39 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:26 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.50:8000/health": dial tcp 10.133.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:37 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:53 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.50:8000/health": dial tcp 10.133.0.50:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:50 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:50 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:50 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:05 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.47:8000/health": dial tcp 10.133.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:25 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:40 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.47:8000/health": dial tcp 10.133.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-5446b4db4b-x7nl4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:35 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-5446b4db4b-2zt2p [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6d64d9cf6c-nwkqq to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:37 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-6d64d9cf6c-znfc4 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:46 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:47 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:47 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:47 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:26 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:26 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-6d64d9cf6c-znfc4 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-6d64d9cf6c-nwkqq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-6d64d9cf6c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-6d64d9cf6c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-5446b4db4b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:35 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-5446b4db4b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:33 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:20:36 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [stop-feature-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:17:27 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [stop-feature-test] is no longer Ready because of: GatewaysReady, HTTPRoutesReady, InferencePoolReady, MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9 to ip-10-0-129-157.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:38 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:38 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:38 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:34 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.133.0.56:8000/health": dial tcp 10.133.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:11 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:27 kserve-ci-e2e-test kubelet/ip-10-0-129-157.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.56:8000/health": dial tcp 10.133.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:33 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-c5d84c7bb-8mkg9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5c94c9d987-lnnd7 to ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.69/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:5b65173770308fe15aad6e4df3359ab251f5bea4d39d3b1da002eeb1d533adbb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:36 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:36 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:36 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:11 kserve-ci-e2e-test kubelet/ip-10-0-132-105.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-router-scheduler-5c94c9d987-lnnd7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-router-scheduler-5c94c9d987 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:33 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-c5d84c7bb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:31 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/tls-verification-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/tls-verification-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/tls-verification-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/tls-verification-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:28:39 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:30:57 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [tls-verification-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-09 10:31:11 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 Flag --decoder-use-tls has been deprecated, use --enable-tls=decoder instead [e2e-llm-inference-service] Flag --prefiller-use-tls has been deprecated, use --enable-tls=prefiller instead [e2e-llm-inference-service] {"level":"info","ts":1783593571.5071177,"msg":"Proxy starting","Built on":"v0.9.0-rc.2","From Git SHA":"181aa8358916e19b8844ccc752b2d6153d4b2ad6"} [e2e-llm-inference-service] {"level":"info","ts":1783593571.5071588,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"kserve-ci-e2e-test\",\"InferencePoolName\":\"scheduler-v06-pd-migration-test-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1783593571.5077436,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"kserve-ci-e2e-test","poolName":"scheduler-v06-pd-migration-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1783593571.5212193,"logger":"allowlist-validator","msg":"InferencePool added","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593571.5299523,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593571.5300055,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593571.6074057,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593571.6080656,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1783593571.6096308,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1783593571.6096427,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1783593572.6150193,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593580.4987712,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593580.5135005,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593580.5844383,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593580.6633694,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593581.6772618,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593601.5216525,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593601.5304139,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593601.5304706,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:40:08 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593608.5056386,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"8e159904-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:40:08 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593608.525316,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"8e18bffe-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:40:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593612.5449386,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"907e0764-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:40:20 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593620.5660388,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"9545f455-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783593631.5225925,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593631.5354564,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593631.5355086,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:40:36 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593636.615319,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"9ed6c4f6-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783593661.522622,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593661.5331829,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593661.5332394,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:41:08 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593668.6395683,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"b1ed5735-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783593691.523449,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593691.533269,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593691.5333247,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593721.524526,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593721.5337327,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593721.5337896,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:42:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593732.6887534,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"d81a709c-7b82-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783593751.5250995,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593751.5347924,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593751.5348587,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593781.5258243,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593781.5351148,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593781.535175,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593811.526398,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593811.5359008,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593811.535963,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593841.5266867,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593841.5359077,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593841.5359588,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:44:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593852.742408,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"1fa929db-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783593871.5271852,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593871.5378451,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593871.5379057,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593901.528008,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593901.5360682,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593901.536125,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593931.5286617,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593931.5381892,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593931.538249,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593961.5294452,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593961.5382624,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593961.538312,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:46:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593972.7953992,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"6737d7e1-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:46:17 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593977.9096274,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"6a4456f3-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:46:17 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593977.9270787,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"6a4703af-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:46:21 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593981.946328,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"6cac3ca4-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:46:29 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783593989.9649477,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"7173d019-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783593991.5301297,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783593991.5402384,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783593991.5402975,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:46:45 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594005.987431,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"7b007f73-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594021.5304985,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594021.5395083,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594021.5395632,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:47:18 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594038.011237,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"8e16a5f7-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594051.5309412,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594051.540675,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594051.5407364,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594081.5311503,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594081.5401351,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594081.540187,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:48:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594102.0582669,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"b443ccea-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594111.531723,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594111.5415077,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594111.5415626,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594141.531876,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594141.5401008,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594141.540159,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594171.532726,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594171.5416992,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594171.5417507,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594201.5335944,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594201.541772,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594201.5418556,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:50:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594222.106871,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"fbd1b053-7b83-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594231.5343215,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594231.5437236,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594231.5437744,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594261.535169,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594261.5449421,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594261.5450165,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594291.5361843,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594291.5456893,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594291.5457504,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594321.5366626,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594321.5449603,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594321.5450113,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:52:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594342.1655586,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"4360bc78-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:52:27 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594347.2524617,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"46698f02-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:52:27 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594347.2695482,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"466c3585-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:52:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594351.2878258,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"48d153d1-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594351.5367644,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594351.5451522,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594351.5452044,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:52:39 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594359.3065739,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"4d98e104-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/09 10:52:55 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594375.3284245,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"572581a4-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594381.537299,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594381.5451722,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594381.5452304,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:53:27 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594407.3513453,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"6a3bd3ef-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594411.537581,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594411.5467503,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594411.5468142,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594441.5386338,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594441.546444,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594441.546499,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:54:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594471.397747,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"90687c26-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594471.5392442,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594471.5473497,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594471.5473967,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594501.5398571,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594501.5479028,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594501.5479608,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594531.5405707,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594531.5502934,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594531.5503478,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594561.5413487,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594561.5497317,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594561.5498176,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:56:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594591.4549956,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"d7f7c1a4-7b84-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594591.542217,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594591.551195,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594591.5512567,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594621.5429692,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594621.550839,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594621.5509048,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594651.5432484,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594651.5520496,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594651.5521064,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594681.5442913,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594681.5521042,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594681.5521529,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] 2026/07/09 10:58:31 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.134.0.43 [e2e-llm-inference-service] {"level":"error","ts":1783594711.4993515,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"1f851b38-7b85-11f1-aee2-0a580a84004e","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1783594711.5451396,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1783594711.5531204,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] {"level":"info","ts":1783594711.5531695,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.132.0.78":{},"10.134.0.43":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn":{}}} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0709 10:39:31.668897 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8001, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1783593571668478500, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0709 10:39:31.703495 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0709 10:39:31.708589 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0709 10:39:31.708647 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0709 10:39:31.708654 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0709 10:39:31.708936 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0709 10:39:31.708950 1 grpc.go:126] "Server starting" protocol="gRPC" port=8001 [e2e-llm-inference-service] I0709 10:39:31.709540 1 http.go:96] "Server starting" protocol="HTTPS" port=8001 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0709 10:39:30.715666 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1783593570715232800, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0709 10:39:30.748991 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0709 10:39:30.753283 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0709 10:39:30.753523 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0709 10:39:30.753745 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0709 10:39:30.753840 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0709 10:39:30.754135 1 grpc.go:126] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0709 10:39:30.754650 1 http.go:96] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] I0709 10:40:08.505558 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:43188\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:40:08.525465 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:43204\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:40:12.544913 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:43218\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:40:20.566030 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:34856\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:40:36.615313 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:49646\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:41:08.639560 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:58670\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:42:12.688726 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:48914\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:44:12.742383 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:48586\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:46:12.795372 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:57166\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:46:17.909605 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:55062\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:46:17.927041 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:55068\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:46:21.946307 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:55080\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:46:29.964925 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:42410\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:46:45.987398 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:53244\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:47:18.011244 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:36296\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:48:22.058255 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:41764\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:50:22.106850 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:50844\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:52:22.165560 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:60908\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:52:27.252412 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:33010\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:52:27.269519 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:33022\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:52:31.287764 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:33028\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:52:39.306714 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:59540\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:52:55.328381 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:57964\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:53:27.351315 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:55660\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:54:31.397719 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:53236\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:56:31.454982 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:57468\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0709 10:58:31.499331 1 communication.go:105] "Server error" msg="error when serving connection \"10.134.0.43:8000\"<->\"10.132.0.78:47314\": remote error: tls: bad certificate" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 ] [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] { [e2e-llm-inference-service] "Name": "gateway.request_orchestration", [e2e-llm-inference-service] "SpanContext": { [e2e-llm-inference-service] "TraceID": "d4363e054ec9f70607bb1203d9613b9c", [e2e-llm-inference-service] "SpanID": "62036e52a0a04870", [e2e-llm-inference-service] "TraceFlags": "01", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "Parent": { [e2e-llm-inference-service] "TraceID": "d4363e054ec9f70607bb1203d9613b9c", [e2e-llm-inference-service] "SpanID": "0fef376ee68c0914", [e2e-llm-inference-service] "TraceFlags": "01", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "SpanKind": 2, [e2e-llm-inference-service] "StartTime": "2026-07-09T10:46:17.913370465Z", [e2e-llm-inference-service] "EndTime": "2026-07-09T10:46:17.913501188Z", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "target_model", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "facebook/opt-125m" [e2e-llm-inference-service] } [e2e-llm-inference-service] }, [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "request_prio", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "INT64", [e2e-llm-inference-service] "Value": 0 [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ], [e2e-llm-inference-service] "Events": null, [e2e-llm-inference-service] "Links": null, [e2e-llm-inference-service] "Status": { [e2e-llm-inference-service] "Code": "Unset", [e2e-llm-inference-service] "Description": "" [e2e-llm-inference-service] }, [e2e-llm-inference-service] "DroppedAttributes": 0, [e2e-llm-inference-service] "DroppedEvents": 0, [e2e-llm-inference-service] "DroppedLinks": 0, [e2e-llm-inference-service] "ChildSpanCount": 5, [e2e-llm-inference-service] "Resource": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.name", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "llm-d-epp" [e2e-llm-inference-service] } [e2e-llm-inference-service] }, [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.version", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "v0.9.0-rc.2" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ], [e2e-llm-inference-service] "InstrumentationScope": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/requestcontrol", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] }, [e2e-llm-inference-service] "InstrumentationLibrary": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/requestcontrol", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] { [e2e-llm-inference-service] "Name": "gateway.request", [e2e-llm-inference-service] "SpanContext": { [e2e-llm-inference-service] "TraceID": "d4363e054ec9f70607bb1203d9613b9c", [e2e-llm-inference-service] "SpanID": "0fef376ee68c0914", [e2e-llm-inference-service] "TraceFlags": "01", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "Parent": { [e2e-llm-inference-service] "TraceID": "00000000000000000000000000000000", [e2e-llm-inference-service] "SpanID": "0000000000000000", [e2e-llm-inference-service] "TraceFlags": "00", [e2e-llm-inference-service] "TraceState": "", [e2e-llm-inference-service] "Remote": false [e2e-llm-inference-service] }, [e2e-llm-inference-service] "SpanKind": 2, [e2e-llm-inference-service] "StartTime": "2026-07-09T10:46:17.913311701Z", [e2e-llm-inference-service] "EndTime": "2026-07-09T10:46:17.927910442Z", [e2e-llm-inference-service] "Attributes": null, [e2e-llm-inference-service] "Events": null, [e2e-llm-inference-service] "Links": null, [e2e-llm-inference-service] "Status": { [e2e-llm-inference-service] "Code": "Unset", [e2e-llm-inference-service] "Description": "" [e2e-llm-inference-service] }, [e2e-llm-inference-service] "DroppedAttributes": 0, [e2e-llm-inference-service] "DroppedEvents": 0, [e2e-llm-inference-service] "DroppedLinks": 0, [e2e-llm-inference-service] "ChildSpanCount": 1, [e2e-llm-inference-service] "Resource": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.name", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "llm-d-epp" [e2e-llm-inference-service] } [e2e-llm-inference-service] }, [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "service.version", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "v0.9.0-rc.2" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ], [e2e-llm-inference-service] "InstrumentationScope": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/handlers", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] }, [e2e-llm-inference-service] "InstrumentationLibrary": { [e2e-llm-inference-service] "Name": "llm-d-router/pkg/epp/handlers", [e2e-llm-inference-service] "Version": "v0.9.0-rc.2", [e2e-llm-inference-service] "SchemaURL": "", [e2e-llm-inference-service] "Attributes": [ [e2e-llm-inference-service] { [e2e-llm-inference-service] "Key": "commit-sha", [e2e-llm-inference-service] "Value": { [e2e-llm-inference-service] "Type": "STRING", [e2e-llm-inference-service] "Value": "181aa8358916e19b8844ccc752b2d6153d4b2ad6" [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] ] [e2e-llm-inference-service] } [e2e-llm-inference-service] } [e2e-llm-inference-service] {"level":"info","ts":1783593981.9313283,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"f0986fce-b07f-41c0-832d-8936cf79c1bf"} [e2e-llm-inference-service] {"level":"info","ts":1783593981.9316523,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"f0986fce-b07f-41c0-832d-8936cf79c1bf","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783593989.9502816,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"9a0bdb8e-a697-4389-a6ff-d8b97e4d8225"} [e2e-llm-inference-service] {"level":"info","ts":1783593989.9506156,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"9a0bdb8e-a697-4389-a6ff-d8b97e4d8225","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594005.9717753,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"9dbfd0d9-1496-495e-b1e4-6ca8dfefe535"} [e2e-llm-inference-service] {"level":"info","ts":1783594005.9721682,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"9dbfd0d9-1496-495e-b1e4-6ca8dfefe535","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594037.993748,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"20b6ea50-d5c1-480d-8b33-6d5becbd7d58"} [e2e-llm-inference-service] {"level":"info","ts":1783594037.9940429,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"20b6ea50-d5c1-480d-8b33-6d5becbd7d58","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594102.043051,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"f27b4735-c053-44fb-a610-b14630f633aa"} [e2e-llm-inference-service] {"level":"info","ts":1783594102.0433536,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"f27b4735-c053-44fb-a610-b14630f633aa","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594222.090087,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"a18e7cd3-d6fb-4e1f-8580-2d64ee8064a2"} [e2e-llm-inference-service] {"level":"info","ts":1783594222.0904372,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"a18e7cd3-d6fb-4e1f-8580-2d64ee8064a2","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594342.1467803,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"0c7dd7e3-99d6-44a8-ade2-b998064fe165"} [e2e-llm-inference-service] {"level":"info","ts":1783594342.1471124,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"0c7dd7e3-99d6-44a8-ade2-b998064fe165","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594347.2377641,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"7dd1184b-3a31-44b6-b931-d99ecbf02457"} [e2e-llm-inference-service] {"level":"info","ts":1783594347.2380536,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"7dd1184b-3a31-44b6-b931-d99ecbf02457","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594347.2557778,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"fb0df87a-b636-4c45-85eb-a7fe98243980"} [e2e-llm-inference-service] {"level":"info","ts":1783594347.2560556,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"fb0df87a-b636-4c45-85eb-a7fe98243980","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594351.2733285,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"a454d203-d3ea-4d4f-bb4f-0a3fa9b178ce"} [e2e-llm-inference-service] {"level":"info","ts":1783594351.273608,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"a454d203-d3ea-4d4f-bb4f-0a3fa9b178ce","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594359.2919235,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"ea45caa1-f442-414b-9fee-6296d5279d9c"} [e2e-llm-inference-service] {"level":"info","ts":1783594359.2922068,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"ea45caa1-f442-414b-9fee-6296d5279d9c","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594375.3129957,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"3eec47a1-2c13-4280-8a42-0fbd32287dcf"} [e2e-llm-inference-service] {"level":"info","ts":1783594375.3132858,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"3eec47a1-2c13-4280-8a42-0fbd32287dcf","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594407.3359985,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"a8856ff4-bae7-42b3-bcef-1ef7dbb1f9bf"} [e2e-llm-inference-service] {"level":"info","ts":1783594407.3362644,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"a8856ff4-bae7-42b3-bcef-1ef7dbb1f9bf","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594471.3821332,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"84049275-8bca-481e-9b30-6bc0ef575906"} [e2e-llm-inference-service] {"level":"info","ts":1783594471.3824303,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"84049275-8bca-481e-9b30-6bc0ef575906","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594591.4365957,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"3a7c3e89-dc49-4cae-82da-8a7c18505e38"} [e2e-llm-inference-service] {"level":"info","ts":1783594591.4369576,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"3a7c3e89-dc49-4cae-82da-8a7c18505e38","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1783594711.4839907,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"d8a7f4e8-5d11-47a0-9122-82ce07a53825"} [e2e-llm-inference-service] {"level":"info","ts":1783594711.484264,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"d8a7f4e8-5d11-47a0-9122-82ce07a53825","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cd748a92-ff46-4f2a-938b-bd9872c0d3cc [e2e-llm-inference-service] resourceVersion: '70773' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.79 [e2e-llm-inference-service] nodeName: ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh [e2e-llm-inference-service] uid: cf6483ce-a2c5-44ec-bc0e-1e7821f6b49e [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 73fc5a30-c724-4493-a5e0-401e4269b41d [e2e-llm-inference-service] resourceVersion: '70251' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.78 [e2e-llm-inference-service] nodeName: ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm [e2e-llm-inference-service] uid: b482f622-c78c-48a8-a968-991a986c44a3 [e2e-llm-inference-service] - ip: 10.134.0.43 [e2e-llm-inference-service] nodeName: ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn [e2e-llm-inference-service] uid: 97122a24-a7d1-41b4-81e7-d16c92891c22 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-5c87756974- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b482f622-c78c-48a8-a968-991a986c44a3 [e2e-llm-inference-service] resourceVersion: '70250' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.78/23"],"mac_address":"0a:58:0a:84:00:4e","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.78/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.78\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:4e\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974 [e2e-llm-inference-service] uid: daf762de-4c9d-4223-8eca-e9b278dfc036 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-105 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"daf762de-4c9d-4223-8eca-e9b278dfc036"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.78"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-krt6h [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-krt6h [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-krt6h [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-2kxwq [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] hostIP: 10.0.132.105 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.105 [e2e-llm-inference-service] podIP: 10.132.0.78 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.78 [e2e-llm-inference-service] startTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-disagg-sidecar@sha256:1ec2c035216ea0a4f8a6f7ccda55398fc3d9f71f487c135f49ba0dea08163f12 [e2e-llm-inference-service] containerID: cri-o://147a11767cc1ba12737d274e18c7057f030c838b43e1e5871a14986e267fcb9e [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-krt6h [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://12c3af9c47870887bce55b4221f4665c093174adc8c41ff500472865eccb06ec [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-krt6h [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 97122a24-a7d1-41b4-81e7-d16c92891c22 [e2e-llm-inference-service] resourceVersion: '70225' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.43/23"],"mac_address":"0a:58:0a:86:00:2b","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.43/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.43\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:2b\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 [e2e-llm-inference-service] uid: 38232440-067d-417c-b8a3-8b686446de21 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-128-170 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"38232440-067d-417c-b8a3-8b686446de21"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.43"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-2lw8k [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-2lw8k [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] hostIP: 10.0.128.170 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.128.170 [e2e-llm-inference-service] podIP: 10.134.0.43 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.43 [e2e-llm-inference-service] startTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://708ea63bdaf929c7590265ca003caa01463f3b2eea86833dccf31497c0337de2 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-2lw8k [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: cf6483ce-a2c5-44ec-bc0e-1e7821f6b49e [e2e-llm-inference-service] resourceVersion: '70772' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.79/23"],"mac_address":"0a:58:0a:84:00:4f","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.79/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.79\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:4f\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd [e2e-llm-inference-service] uid: 0431d62a-b887-49ed-99f1-4e91a96d6458 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-105 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0431d62a-b887-49ed-99f1-4e91a96d6458"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.79"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-pq9jb [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name:\ [e2e-llm-inference-service] \ prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-pq9jb [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000700000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c25 [e2e-llm-inference-service] fsGroup: 1000700000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-8zthl [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] hostIP: 10.0.132.105 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.105 [e2e-llm-inference-service] podIP: 10.132.0.79 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.79 [e2e-llm-inference-service] startTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://65e34559ba3e0c3d7076df9ce4d621ce640b32f2748c2da505086c723e2c7bc4 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-pq9jb [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000700000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000700000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4ae908ab-1a8d-473f-8448-52c30ce69269 [e2e-llm-inference-service] resourceVersion: '69910' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-epp-sa-dockercfg-8zthl [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-epp-sa-dockercfg-8zthl"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-vkhz8"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-8zthl [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-8zthl [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: f195e24a-64f3-41ac-a9ab-bf5e52560585 [e2e-llm-inference-service] resourceVersion: '69872' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-kserve-dockercfg-2kxwq [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-kserve-dockercfg-2kxwq"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-vkhz8"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-2kxwq [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-vkhz8 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-2kxwq [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 22637fe5-dd3d-40dc-9cbf-6b70bee873dc [e2e-llm-inference-service] resourceVersion: '69934' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.85.180 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.85.180 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a98466b7-aceb-4c90-aa74-4706aaff2cd8 [e2e-llm-inference-service] resourceVersion: '69891' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.185.249 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.185.249 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b3d7c584-7310-4747-b466-86d255210a4a [e2e-llm-inference-service] resourceVersion: '70256' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-5c87756974" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: a90fa950-4812-4caa-9366-a43784991554 [e2e-llm-inference-service] resourceVersion: '70231' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 3ca10692-0021-4037-a8fb-ae5af06a4e3a [e2e-llm-inference-service] resourceVersion: '70776' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n\ [e2e-llm-inference-service] \ prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: daf762de-4c9d-4223-8eca-e9b278dfc036 [e2e-llm-inference-service] resourceVersion: '70255' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] uid: b3d7c584-7310-4747-b466-86d255210a4a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"b3d7c584-7310-4747-b466-86d255210a4a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 38232440-067d-417c-b8a3-8b686446de21 [e2e-llm-inference-service] resourceVersion: '70230' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] uid: a90fa950-4812-4caa-9366-a43784991554 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a90fa950-4812-4caa-9366-a43784991554"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:40Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0431d62a-b887-49ed-99f1-4e91a96d6458 [e2e-llm-inference-service] resourceVersion: '70775' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 3ca10692-0021-4037-a8fb-ae5af06a4e3a [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3ca10692-0021-4037-a8fb-ae5af06a4e3a"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n\ [e2e-llm-inference-service] \ prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8a16b9e2-0b72-4636-b356-94324d380a13 [e2e-llm-inference-service] resourceVersion: '69928' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 63b136e3-f51b-46d6-b1cb-bad3c7adec52 [e2e-llm-inference-service] resourceVersion: '69878' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d959cf96-c8db-45f1-a870-e0014fc65d06 [e2e-llm-inference-service] resourceVersion: '69926' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: df4bd798-07d8-4c32-bebf-3fbec47ea77f [e2e-llm-inference-service] resourceVersion: '69876' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service-hl25z [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7a2752c0-d7cb-4bfe-b002-a19e481560dd [e2e-llm-inference-service] resourceVersion: '70774' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] uid: 22637fe5-dd3d-40dc-9cbf-6b70bee873dc [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:40:02Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"22637fe5-dd3d-40dc-9cbf-6b70bee873dc"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.79 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh [e2e-llm-inference-service] uid: cf6483ce-a2c5-44ec-bc0e-1e7821f6b49e [e2e-llm-inference-service] nodeName: ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc-jqfp7 [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 69f01aee-f964-44a8-889a-67fea52aba6f [e2e-llm-inference-service] resourceVersion: '70254' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] uid: a98466b7-aceb-4c90-aa74-4706aaff2cd8 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:41Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"a98466b7-aceb-4c90-aa74-4706aaff2cd8"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.43 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn [e2e-llm-inference-service] uid: 97122a24-a7d1-41b4-81e7-d16c92891c22 [e2e-llm-inference-service] nodeName: ip-10-0-128-170.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.78 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm [e2e-llm-inference-service] uid: b482f622-c78c-48a8-a968-991a986c44a3 [e2e-llm-inference-service] nodeName: ip-10-0-132-105.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 8a16b9e2-0b72-4636-b356-94324d380a13 [e2e-llm-inference-service] resourceVersion: '69928' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 63b136e3-f51b-46d6-b1cb-bad3c7adec52 [e2e-llm-inference-service] resourceVersion: '69878' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: d959cf96-c8db-45f1-a870-e0014fc65d06 [e2e-llm-inference-service] resourceVersion: '69926' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: df4bd798-07d8-4c32-bebf-3fbec47ea77f [e2e-llm-inference-service] resourceVersion: '69876' [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-09T10:39:29Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70196' [e2e-llm-inference-service] uid: 938815ca-43a5-49c3-8b2d-270825eaf960 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70196' [e2e-llm-inference-service] uid: 938815ca-43a5-49c3-8b2d-270825eaf960 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70182' [e2e-llm-inference-service] uid: 2a7375ae-d4d3-456a-aa3b-a6bbe37d1c0d [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:38Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:32Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:32Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-09T10:39:34Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70112' [e2e-llm-inference-service] uid: a5cd3005-af4a-46d6-91b8-1d3260baddca [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:33Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-09T10:39:34Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '69970' [e2e-llm-inference-service] uid: cab7cce7-6d2a-4b46-824d-56ac5056eaa5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70191' [e2e-llm-inference-service] uid: 8f9e911b-73b6-4378-afdd-bc93aecf5d81 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70018' [e2e-llm-inference-service] uid: 51f63789-0325-4d0f-a75d-41047c8041b4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '69970' [e2e-llm-inference-service] uid: cab7cce7-6d2a-4b46-824d-56ac5056eaa5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70191' [e2e-llm-inference-service] uid: 8f9e911b-73b6-4378-afdd-bc93aecf5d81 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70018' [e2e-llm-inference-service] uid: 51f63789-0325-4d0f-a75d-41047c8041b4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '69970' [e2e-llm-inference-service] uid: cab7cce7-6d2a-4b46-824d-56ac5056eaa5 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:39Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70191' [e2e-llm-inference-service] uid: 8f9e911b-73b6-4378-afdd-bc93aecf5d81 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:31Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '70018' [e2e-llm-inference-service] uid: 51f63789-0325-4d0f-a75d-41047c8041b4 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"3c92656e-fb2a-4834-bf2c-a95b429b6fdf"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-09T10:39:30Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 3c92656e-fb2a-4834-bf2c-a95b429b6fdf [e2e-llm-inference-service] resourceVersion: '69944' [e2e-llm-inference-service] uid: 66540ba8-7597-48b1-9a6d-f8e62edc94c7 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:58:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] timestamp: '2026-07-09T10:58:16Z' [e2e-llm-inference-service] window: 19.821s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16582563n [e2e-llm-inference-service] memory: 24988Ki [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 9181598n [e2e-llm-inference-service] memory: 20328Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:58:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] timestamp: '2026-07-09T10:58:13Z' [e2e-llm-inference-service] window: 10.215s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 18121488n [e2e-llm-inference-service] memory: 25116Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-09T10:58:33Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] timestamp: '2026-07-09T10:58:23Z' [e2e-llm-inference-service] window: 16.612s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16510233n [e2e-llm-inference-service] memory: 32800Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-09T10:58:34.006973] end - ❌ 1166.505s: Service returned 502: [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] ====== 1 failed, 41 passed, 3 skipped, 29 warnings in 4339.23s (1:12:19) ======= [must-gather] [must-gather ] OUT 2026-07-09T10:58:35.679097626Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 99737322-a04c-4b9b-8075-44dafcd2cd09 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T10:58:35.734694944Z namespace/openshift-must-gather-c4n8h created [must-gather] [must-gather ] OUT 2026-07-09T10:58:35.740441437Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-ml2zb created [must-gather] [must-gather ] OUT 2026-07-09T10:58:35.772873713Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.472808816Z [disk usage checker] Started [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.476177039Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.742144793Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.909525445Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.909558557Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.909558557Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:42.912511267Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.080574949Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.080607125Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.080607125Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.083545433Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.244772297Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.244868780Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.244868780Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.247608709Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.411365568Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.411457498Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.411457498Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.414309690Z Error getting logs from rhods-notebooks [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.578697343Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.578727013Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.578727013Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.581320551Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.750775140Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.750869273Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.750869273Z namespaces "istio-system" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.753693319Z Error getting logs from istio-system [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.913414510Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.913443632Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.913443632Z namespaces "knative-serving" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:43.916336874Z Error getting logs from knative-serving [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.079705759Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.079737426Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.079737426Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.082456620Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.360265745Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.646030608Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.784296871Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.866840787Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:44.869621030Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.012272827Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.091215729Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.093593102Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.229107444Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.311722220Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.314200562Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.448799904Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.533107659Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.535609042Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.673428683Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.756842174Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.759314228Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.893058561Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.978861973Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:45.981551690Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.118710178Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.202953683Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.205945549Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.339607929Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.426297597Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.429069721Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.561142904Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.642192852Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.644821818Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.783665603Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.866434387Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:46.868982449Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.006006565Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.091291131Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.093964437Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.231056214Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.313580006Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.316333092Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.447434449Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.480657593Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.532131930Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.534691889Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.669036417Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.754079998Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.756813552Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.889137539Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.971034150Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:47.973510496Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.106777625Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.186514783Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.188899000Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.323183309Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.408969520Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.411711822Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.543978886Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.628246207Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:48.630806524Z Error collecting info from [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:49.344496202Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:49.475019211Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:49.608082451Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:49.747245249Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:49.880827347Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:50.492209014Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:50.623952771Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:50.765119657Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:50.905131182Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:51.035701219Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:51.586387268Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:51.907069578Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:52.485376899Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:52.629019451Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:52.980138647Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:53.147560297Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:53.437110047Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:54.152721522Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:54.515928081Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:54.680178886Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:54.961513985Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:55.672664826Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:56.046737340Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:56.217726914Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:56.513066316Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:57.224709617Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:57.493985697Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:57.609917058Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:57.782779767Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:58.055224450Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:58.778594572Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:59.127133329Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:59.294558014Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:58:59.586588340Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:00.301155156Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:00.655598860Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:00.824071876Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:01.109335146Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:01.832721772Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:02.210745043Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:02.384174864Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:02.498822923Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:02.681658011Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:03.439469091Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:03.821947654Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:03.995524344Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:04.297456603Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:05.062431521Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:05.449270955Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:05.622199464Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:05.933682494Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:06.650310863Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:07.018455642Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:07.187445218Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:07.471534590Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:07.505803208Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:08.178541803Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:08.552440793Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:08.718792960Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:09.006588500Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:09.718525566Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:10.069154072Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:10.238827553Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:10.532847076Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:11.252754172Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:11.631544131Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:11.803976458Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:12.093668082Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:12.510444155Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:12.794292633Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:13.144850008Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:13.307581886Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:13.585884402Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:14.298988000Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:14.649370711Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:14.813622241Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:15.088732781Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:15.810576491Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:16.180470312Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:16.350456552Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:16.952921222Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:17.123050984Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:17.508280295Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:17.515000790Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:17.681740478Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:18.323699895Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:18.480565448Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:18.864735289Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:19.034091554Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:19.628864613Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:19.787673953Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:20.167452557Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:20.331939361Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:20.913252527Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:21.057125273Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:21.437219557Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:21.603315386Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:22.084280333Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:22.277290633Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:22.519445416Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:22.680306049Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:22.855029205Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:23.329562692Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:23.522521041Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:23.922577663Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:24.088516277Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:24.539098899Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:24.716481126Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:25.116495283Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:25.283965244Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:25.738676186Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:25.922909688Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:26.320466130Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:26.485770274Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:26.957320859Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:27.139554692Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:27.523998533Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:27.543128336Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:27.715058062Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:28.192168553Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:28.371586038Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:28.772218694Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:28.939089177Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:29.410141801Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:29.589092297Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:29.994440453Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:30.163757997Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:30.662731487Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:30.856283838Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:31.252789093Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:31.421548661Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:31.884533077Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:32.065647793Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:32.464330473Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:32.528547265Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:32.631056014Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:33.091544444Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:33.272823134Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:33.670498088Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:33.836260478Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:34.307291571Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:34.491446510Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:34.888637336Z Wrote inspect data to must-gather. [must-gather] [must-gather-h5xzb] POD 2026-07-09T10:59:34.905359091Z Caches written to disk [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:35.914606999Z waiting for gather to complete [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:35.918576505Z downloading gather output [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.228292506Z receiving incremental file list [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.248159915Z ./ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.248247172Z aggregated-discovery-api.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.250798158Z aggregated-discovery-apis.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.251588319Z event-filter.html [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.25420297Z timestamp [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.25434015Z version [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.269854914Z cluster-scoped-resources/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.269872206Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.269878036Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.26992752Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270069641Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270086012Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270129595Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270221363Z namespaces/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270236794Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270250695Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270371734Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270426608Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270489713Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270536957Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.270638935Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271045806Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271422805Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271469759Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271508082Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271565286Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.27161295Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271688105Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271761171Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271802384Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.27187802Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271929534Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.271988118Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.27214168Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.272249639Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.278612948Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.278683434Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279132618Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279259038Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279493786Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279575762Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279623536Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279704732Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279722134Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279778998Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279823831Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.279902557Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.27993507Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280015686Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280077381Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280111504Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280161287Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280173228Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280184879Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/e2e-pvc-model-download-xvtw2.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280270836Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/storage-initializer/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280281497Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/storage-initializer/storage-initializer/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280286737Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/storage-initializer/storage-initializer/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.280298088Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/storage-initializer/storage-initializer/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285243518Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/storage-initializer/storage-initializer/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285307493Z namespaces/kserve-ci-e2e-test/pods/e2e-pvc-model-download-xvtw2/storage-initializer/storage-initializer/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285325435Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285377029Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/router-gateway-1-openshift-default-b49948d49-g6vgn.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285489817Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/istio-proxy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285506429Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/istio-proxy/istio-proxy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285513219Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28552703Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285689933Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-b49948d49-g6vgn/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285773579Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285796191Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/router-gateway-2-openshift-default-578d7b9c-m7zq9.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285905799Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/istio-proxy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28591412Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/istio-proxy/istio-proxy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28591839Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.285928211Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286081883Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-578d7b9c-m7zq9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286164799Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286204442Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286295889Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/llm-d-routing-sidecar/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28630331Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28630765Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286333262Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286593472Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286660328Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286677959Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/main/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28669161Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/main/main/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286705071Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/main/main/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286724052Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/main/main/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286852862Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/main/main/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286922548Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-4xxsm/main/main/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.286973421Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287033686Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287113982Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/main/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287121183Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/main/main/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287125543Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/main/main/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287147325Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/main/main/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287273925Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/main/main/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28734433Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-vghgn/main/main/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287384983Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287413585Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287499312Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/main/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287506702Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/main/main/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287511153Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/main/main/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287535625Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/main/main/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287803495Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/main/main/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287871781Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59wqtrh/main/main/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287901583Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.287930925Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288011331Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288053225Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288102278Z namespaces/kuadrant-system/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288136621Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288219717Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288262301Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288296023Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288346287Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288429784Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.288725776Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289021619Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289046801Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289110926Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.28915836Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289170001Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289269508Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289315902Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289351575Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289438701Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289475364Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289512667Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289675689Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.289788498Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.290210561Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.290284626Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.290638074Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.290745222Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.290968299Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291098749Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291130221Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.29124224Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291276152Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291315926Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291362049Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291447486Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291487529Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291508481Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291568445Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291599617Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291641711Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291648661Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291683754Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino-7c77957575-z7vd9.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291770761Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291786322Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291791912Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.291804823Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.294892151Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.294959666Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.29501526Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295046103Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/authorino-operator-6d85f6564-xhvv8.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.29514531Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295153921Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295160791Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295170532Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295369978Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295437143Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295479446Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295515419Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/dns-operator-controller-manager-65b49595d7-qm9ts.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295608206Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295615877Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295619807Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295634448Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295751317Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295819162Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295842374Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295891358Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin-85fd4b8b9d-2ndhv.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295973234Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295986385Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.295992015Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296031338Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296105384Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296173249Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296212323Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296232714Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296366834Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296377445Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296383546Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.296393566Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304344598Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304404423Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304433095Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304477728Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador-limitador-69574b596d-9r2d7.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304567585Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304578466Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304582676Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.30462177Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304707186Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304769561Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304801023Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304851177Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/limitador-operator-controller-manager-6f9f468797-g7cbp.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304937494Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304969766Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304974837Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.304983697Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.3051535Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305222396Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305243147Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305295501Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305345085Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305369437Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305435612Z namespaces/openshift-ingress/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305458204Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.30553673Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305559751Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305610955Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305651199Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305737415Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.305969953Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306202991Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306222663Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306284927Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306369064Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306387875Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306494474Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306528786Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306557868Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306649735Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306685958Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.306721821Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.307175306Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.307266713Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.307471179Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.307545964Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.307787853Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.307913053Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308197365Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308284161Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308306913Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308411851Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308431763Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308500508Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308518449Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308618267Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.30865883Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308677102Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308753997Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308768679Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308828863Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308838624Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308861196Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/istiod-openshift-gateway-94bb8fbfd-m2d5h.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308977815Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308988666Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.308993156Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.309024278Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.33296457Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333029805Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333063028Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.33309565Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/openshift-ai-inference-openshift-default-655f7c58b4-962b9.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333242392Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333251262Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333256403Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333284015Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.333843198Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.33400305Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334025532Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router-default-c7f6c5c4f-cjdpd.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334142091Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334149962Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334154252Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334193615Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/current.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334296513Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/previous.insecure.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334361738Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/previous.log [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334401511Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334444784Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334486177Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.334527431Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.347725386Z [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.347743268Z sent 3,435 bytes received 1,338,589 bytes 894,682.67 bytes/sec [must-gather] [must-gather-h5xzb] OUT 2026-07-09T10:59:36.347748768Z total size is 16,649,793 speedup is 12.41 [must-gather] [must-gather ] OUT 2026-07-09T10:59:36.597408869Z namespace/openshift-must-gather-c4n8h deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 99737322-a04c-4b9b-8075-44dafcd2cd09 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T10:59:36.722557088Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba272a129f91bab6ed70a10b041d49836b63538fa2cbaa919e53a4a13fe3dcde [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 99737322-a04c-4b9b-8075-44dafcd2cd09 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-09T10:59:36.747211135Z namespace/openshift-must-gather-d8w7l created [must-gather] [must-gather ] OUT 2026-07-09T10:59:36.755370083Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-qx4cx created [must-gather] [must-gather ] OUT 2026-07-09T10:59:36.783099797Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba272a129f91bab6ed70a10b041d49836b63538fa2cbaa919e53a4a13fe3dcde created [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:38.125507418Z [disk usage checker] Started [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:38.129133149Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:38.334591672Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:38.487789010Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:38.590155043Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:38.693676098Z Gathering data for ns/default... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:39.031639159Z Gathering data for ns/openshift... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:39.489422800Z Gathering data for ns/kube-system... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:40.266615990Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:40.636973858Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:40.637007729Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:40.637007729Z namespaces "assisted-installer" not found [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.082820420Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.087216857Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.092144771Z INFO: Gathering HAProxy config files [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.094217815Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.102340171Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.103012800Z Executing Istio gather script [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.152320909Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.175623924Z INFO: Collecting host service logs for crio [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.176016590Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.176379409Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.176703243Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.177043802Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.177382035Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.177707506Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.178039412Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.178368294Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.178693577Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.179024714Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.179404126Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:41.924359469Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.203593469Z INFO: Collecting Insights Archives from insights-operator-5c847dcfc4-d5888 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.203593469Z insights-runtime-extractor-7hkrk [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.203593469Z insights-runtime-extractor-sn8f2 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.203593469Z insights-runtime-extractor-xvqd8 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.303686239Z No resources found [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.333012858Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.390514608Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.390596312Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.394321922Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.419536255Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.541316895Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.950271190Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:42.994000900Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.356188922Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.451331844Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.486456789Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.486456789Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.547249604Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.670201795Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.815696427Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.898187360Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.938625982Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.958268754Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.965058092Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:43.990787427Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.029671374Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.051245136Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.097540853Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.127058156Z No resources found [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.201061211Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.205729410Z INFO: HAProxy config collection complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.354660085Z INFO: 'previous-logs' folder not found on ip-10-0-128-170.ec2.internal, skipping... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.388552487Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.604224943Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.646393520Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.824685066Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.835476842Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.871531813Z INFO: 'previous-logs' folder not found on ip-10-0-129-157.ec2.internal, skipping... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.885633690Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:44.889263051Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.075343712Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.277648137Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.432075050Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.435853361Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.460574436Z INFO: 'previous-logs' folder not found on ip-10-0-132-105.ec2.internal, skipping... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.460764490Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.460817940Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.468783284Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.574827165Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.695423211Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.696605190Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.708474496Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.875098890Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.890570993Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:45.899932463Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.082356336Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.204056502Z INFO: INTERCONNECT MODE [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.204151158Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.282868402Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.290862368Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.298327168Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.332677779Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.370429603Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.481004932Z INFO: Gathering OVN_Northbound from ovnkube-node-4jtwj... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.481264471Z INFO: Gathering OVN_Northbound from ovnkube-node-v8sxk... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.481558395Z INFO: Gathering OVN_Northbound from ovnkube-node-vmx6b... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.481871135Z INFO: Gathering OVN_Southbound from ovnkube-node-4jtwj... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.482187449Z INFO: Gathering OVN_Southbound from ovnkube-node-v8sxk... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.482480535Z INFO: Gathering OVN_Southbound from ovnkube-node-vmx6b... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.720500082Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.726688771Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.735587205Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.782666735Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.821351685Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.846226476Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.872869404Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.892702348Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.905078553Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:46.959122100Z tar: Removing leading `/' from member names [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.132493273Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.204213005Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.230594156Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.545881648Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.546232761Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.546851893Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.548868072Z 7.4M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.551456671Z ovnk_database_store/ [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.551456671Z ovnk_database_store/ovnkube-node-4jtwj_nbdb [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.556655905Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.565508124Z ovnk_database_store/ovnkube-node-vmx6b_sbdb [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.578254189Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.609499857Z ovnk_database_store/ovnkube-node-v8sxk_nbdb [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.626247676Z ovnk_database_store/ovnkube-node-vmx6b_nbdb [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.642130116Z ovnk_database_store/ovnkube-node-v8sxk_sbdb [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.694336175Z ovnk_database_store/ovnkube-node-4jtwj_sbdb [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.742736591Z INFO: Network log collection complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:47.855316333Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.020374938Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.110386986Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.189905727Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.438910844Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.686180143Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.709073553Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.711657417Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:48.946741577Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.295039323Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.332769090Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5be8cdcf34f35a89a70b6bfa49e66897641871237570b3e09766f83a0a473e6a [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.439772500Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.482023132Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.533611722Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.654305841Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.790064032Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.793196816Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:49.937797764Z Gathering data for ns/openshift-console... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.082436960Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.304534953Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.308448779Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.506003323Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.555586061Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.740794098Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.742958096Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.794149359Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:50.947077240Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.176533910Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.179227949Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.182921262Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.392442983Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.532454293Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.624500994Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.626831387Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.812350379Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:51.945971646Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.047689474Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.050635782Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.228283511Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.269251033Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.488310597Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.491211681Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.730958140Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.791136009Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.958515128Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:52.961690499Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.107211030Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.175567924Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.227396996Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.410397612Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.413339857Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.607179257Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.694760307Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.828532643Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:53.831031089Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.004164319Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.217723506Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.243315754Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.245527485Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.262183808Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.437827930Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.660569131Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.662943069Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.730878880Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:54.844465766Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.046481929Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.049248176Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.237586790Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.410311210Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.485739067Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.489489844Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.665139568Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.867540685Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:55.870186433Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.049131636Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.264239637Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.266614389Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.466676946Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.556374560Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.676759773Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.679265499Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:56.885212993Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.089446003Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.092088998Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.292028434Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.511610945Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.514188585Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.707747278Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.710959343Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.728211052Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.960045450Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:57.963315274Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.019105273Z Collecting performance related data for node ip-10-0-128-170.ec2.internal [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.026269304Z Collecting performance related data for node ip-10-0-129-157.ec2.internal [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.037130412Z Collecting performance related data for node ip-10-0-132-105.ec2.internal [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.247134223Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.495223250Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.498364946Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.525851305Z Gathering data for ns/kserve... [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.704455778Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:58.740948085Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:59.019230681Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:59.022292174Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:59.259514333Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:59.545389419Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:59.549629425Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-78xgd] POD 2026-07-09T10:59:59.809073495Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:00.050773301Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:00.170537691Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:00.393320862Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.001584232Z Gathering data for ns/openshift-config... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.036579128Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.235571797Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.444049565Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.519763075Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.608658759Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:01.820470543Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:02.072563020Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:02.451697558Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:02.480969302Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:02.688014909Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:02.806354357Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:02.947245378Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:03.194622381Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:03.499037142Z Done executing Istio gather script [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:03.597840127Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:03.715606275Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:03.953106058Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:04.368076939Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:04.762506349Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:05.122499808Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:05.713513370Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:06.417427484Z Collecting kubelet logs for node ip-10-0-128-170.ec2.internal [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:06.419129646Z Collecting kubelet logs for node ip-10-0-129-157.ec2.internal [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:06.420822952Z Collecting kubelet logs for node ip-10-0-132-105.ec2.internal [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:06.866069383Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-d8w7l namespace [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:06.872088464Z INFO: Node performance data collection complete. [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:07.315450423Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:08.718516813Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:08.720672827Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:09.093393177Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:09.676280097Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:10.165214904Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:10.580191167Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:10.875251347Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:11.398954496Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:11.728140839Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:12.198718468Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:12.497081320Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:12.840655188Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:13.533068683Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:13.726497625Z [disk usage checker] Volume usage percentage: current = 15 ; allowed = 70 [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:13.849997411Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:14.183903067Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:14.572793284Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:15.566402982Z Wrote inspect data to must-gather. [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:15.566451993Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:15.566451993Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:15.710112686Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-78xgd] POD 2026-07-09T11:00:15.929713062Z Caches written to disk [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:16.943149162Z waiting for gather to complete [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:16.94740298Z downloading gather output [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.255743796Z receiving incremental file list [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.27861939Z ./ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.278756801Z aggregated-discovery-api.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.279179163Z aggregated-discovery-apis.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.280092253Z event-filter.html [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.283344634Z timestamp [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.283439402Z version [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289411012Z cluster-scoped-resources/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289429524Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289443845Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289527221Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289666072Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289794872Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289884059Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.289971145Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.290112516Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.290195623Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.290326633Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.290373076Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.290497276Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.290582852Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.291037267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.29107179Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.291286136Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.291567518Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.291777414Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.291930096Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.295970428Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.297890706Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.298050008Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.298182078Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.298292997Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.298436478Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.29897879Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.299142142Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.299300465Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.299415113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.300244727Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.300628127Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.300775218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.300963243Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.302202689Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.302430846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.302599789Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.302772012Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.30313472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.303275071Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.303475656Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.303744137Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.304717832Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.304852793Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.304993413Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.305118123Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.305218141Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.305328089Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.305490272Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.305743311Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.306580756Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.306718837Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.306862778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.30883709Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.309039905Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.309202668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.309436846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.30974095Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.310193225Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.310311894Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.310436073Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.310546842Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.3106558Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.310799491Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.310903499Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.311089384Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.311227314Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.311334533Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.311803379Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.311961311Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.312118803Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.312689857Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.312815587Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.312934786Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.313159093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.313958615Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.314086185Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.31428057Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.314395878Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.314574552Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.314699252Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.314875135Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315038548Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315162018Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315283757Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315411947Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315542287Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315686228Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315827709Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.315980191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.316146353Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.316292915Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.316684395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.316857408Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.317298462Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.317613917Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.318528267Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.318672838Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.318795798Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.318925698Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.31907529Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.319266154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.319527094Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.319658374Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.319763623Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.319933996Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.320088508Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.320207207Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.320352418Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.3205033Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.32063747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.323001542Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.323470208Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.324290192Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.324472475Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.324644969Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.32478738Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.32492429Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.325059341Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.325165069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.326014874Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.32634482Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.329510684Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.329629323Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.329785395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.331973574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.332120925Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.332265696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.332385116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.33542711Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.335724093Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.342840822Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.347236521Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.347411254Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.347714288Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.347911443Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.348093237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.348224147Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.348406011Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.348682192Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.348811432Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.348956923Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.349160939Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.349310061Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.349443891Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.349580402Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.349714352Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.349873074Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.350054168Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.355378899Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.355541551Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.355744707Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.355968364Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.356125996Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.356314271Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.356448391Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.356592432Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.356873814Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.357035116Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.357291936Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.357428787Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.357695817Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.357818237Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.359904328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.360074961Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.360213882Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.36032532Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.360446069Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.360616013Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.360754633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.360923656Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.361089309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.362645339Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.362844544Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.362987916Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.36317328Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.363326282Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.363525867Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.363782227Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.364327319Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.364620051Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.364763602Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.364891672Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.365035113Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.365672983Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.365888389Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.366049872Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.367503574Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.367648005Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.367795506Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.367898154Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.368155704Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.368430265Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.368552685Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.368710177Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.36913918Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.369264109Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.369392959Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.369555982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.369704053Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.369832773Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.369990845Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370155348Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370489124Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370557329Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370661067Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370762715Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370867163Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.370987032Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371171416Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371294156Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371389093Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371494341Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371591809Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371701567Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371802675Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.371908983Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372034703Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372138121Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372239479Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372351978Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372440074Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372526741Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372613368Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372700224Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372801952Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372887949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.372994547Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373091504Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373173981Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373259868Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373351575Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373435341Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373527828Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373633586Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373748035Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373843793Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.373962962Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374171708Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374270135Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374371583Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374484422Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37458599Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374688978Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374810977Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.374924656Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375069277Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375171775Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375282644Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375388472Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37549782Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37562915Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375734848Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375847127Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.375954025Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376093326Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376198454Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376304132Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37640812Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376523019Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376630068Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376745046Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376852014Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.376962093Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.377108104Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.377205852Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37731208Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37743948Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37756942Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.377677298Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.377786066Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.377888485Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378017374Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378119782Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378229431Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378337119Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378449428Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378557006Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378653524Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378764072Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.378895782Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379041513Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.37913405Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379237129Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379343547Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379449795Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379578375Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379680823Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379791421Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.379897949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380059742Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380195562Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380327993Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38043253Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380546989Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380663928Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380780247Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.380879075Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381004655Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381141435Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381256624Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381374393Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381484832Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381601901Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38172236Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381834879Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.381937097Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382073837Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382189296Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382304165Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38236509Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.3823762Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382427114Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/csr-xw29x.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382566395Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-5559q.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382682364Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-mjh7k.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.382876219Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/system:openshift:openshift-monitoring-qlc6l.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.383048822Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.383108037Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.383287981Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.383445623Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.383557782Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.383687662Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384079772Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384214352Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38431196Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384424638Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384559249Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384652066Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384756234Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.384858962Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385023445Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385127983Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385283995Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385406334Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385497411Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385586408Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385675445Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385764022Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38587444Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.385969737Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.386103898Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38613835Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38626352Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38638713Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.386491498Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.386605446Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.386722406Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.386850576Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387045171Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38716621Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387268828Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387381026Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387492035Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387635636Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387755735Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387857833Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.387990843Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388110723Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388222441Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38833041Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388440768Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388554417Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388661105Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.38872313Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388810167Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388869351Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.388968219Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389026763Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389105689Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389162074Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389228869Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389286443Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389364029Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389422074Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389512771Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389526412Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389573926Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389707736Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389818704Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389884619Z cluster-scoped-resources/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.389901581Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390008029Z cluster-scoped-resources/core/nodes/ip-10-0-128-170.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390205224Z cluster-scoped-resources/core/nodes/ip-10-0-129-157.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390346955Z cluster-scoped-resources/core/nodes/ip-10-0-132-105.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390483076Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390505797Z cluster-scoped-resources/core/persistentvolumes/pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390610515Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390661539Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390702653Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39080056Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.390959752Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391023437Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391146047Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391226713Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391413537Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391507855Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391609352Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391715681Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391801757Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.391894114Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392004673Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39209357Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392182127Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392269063Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39235488Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392441337Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392536114Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392622161Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392707707Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392792764Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392837627Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.392922204Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393021711Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393105388Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393183234Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39325864Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393409011Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.3935282Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393609357Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393670641Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393678512Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393721505Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393791351Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393798601Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393836684Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39391351Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.393966434Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394111435Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394122196Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39417504Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394267748Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394421039Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394473943Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394483774Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394501195Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394582132Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394593542Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394617995Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394701771Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394726043Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394862153Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.394879264Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395031246Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395055678Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395152976Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395174417Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395301537Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395323669Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395436778Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395459429Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395539595Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395566057Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395638623Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395675036Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395753072Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395815387Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395891733Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.395963118Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396014652Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396075427Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396107899Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396158183Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396215198Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396270012Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396394972Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396406912Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396421483Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396515981Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396548153Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396681374Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396807833Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.396997218Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397141589Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397260458Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397387458Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397483145Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39754737Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397583523Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397692381Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39780326Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.397972333Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398136546Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398276917Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398379004Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398471052Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398558078Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398646775Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398736512Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398815838Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.398925467Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399044896Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399127772Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.3992253Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399314297Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399404144Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.39949542Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399580297Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399666814Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399773602Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399816995Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.399916683Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.400019651Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.400111008Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.400257699Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.400556993Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40064868Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40079208Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.400878767Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.400986335Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401072592Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40117073Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401267117Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401352674Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40144047Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401526757Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401615324Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401700341Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401787487Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.401882665Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402035907Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402131584Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40221513Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402302787Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402393074Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40247577Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402562147Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402652844Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402738341Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402843219Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402895043Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402902133Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.402965828Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403029763Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403037114Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403079717Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403142852Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403150342Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403194406Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403259471Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403302844Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-128-170.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403395301Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-129-157.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403473827Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-132-105.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403538992Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403584526Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403665712Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403755749Z host_service_logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403787951Z host_service_logs/masters/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403834845Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403903471Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.403985787Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404057302Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404180412Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404311282Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404387838Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404459683Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404528979Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404602234Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40467989Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404737065Z ingress_controllers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404744775Z ingress_controllers/default/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404759546Z ingress_controllers/default/router-default-c7f6c5c4f-cjdpd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404796499Z ingress_controllers/default/router-default-c7f6c5c4f-cjdpd/haproxy.config [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.404926719Z insights-data/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40493668Z istio/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.405017766Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.405143136Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.405673277Z istio/event-filter.html [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.40584539Z istio/timestamp [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406002102Z istio/cluster-scoped-resources/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406014833Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406053136Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406062677Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406153354Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406192377Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406253041Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406335418Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406383022Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.406549935Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.407482847Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.407697123Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.407872897Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.408292329Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.408452511Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.408840411Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.409684376Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.409833608Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.409964328Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410193465Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410327376Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410452865Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410574515Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410683063Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410795232Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.410961225Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41116081Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.411413519Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.411597504Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412120344Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412270416Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412399215Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412543607Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412556078Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412600651Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412673997Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412684068Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412729131Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412813178Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.412960259Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413062247Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41310493Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413195787Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413348779Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413411073Z istio/namespaces/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413421454Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413443826Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413454227Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41349986Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413589177Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413687375Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413728388Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413823235Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.413904031Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41401049Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414091856Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/scheduler-v06-pd-migration-test-kserve-route.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414215806Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414222546Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414257559Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414329584Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414337175Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414374998Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414443833Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414454844Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414508128Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-scheduler.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414603385Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-shadow-svc.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414773609Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-workload-svc.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414866076Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.414914629Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415001506Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41504865Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415099834Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415143197Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415251205Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415486704Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415708881Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415764705Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415821759Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.415889784Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41596236Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416077229Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416149665Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416178647Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416270924Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416370352Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416417885Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41686822Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.416984999Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.417293313Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.417377239Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.417617668Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.417755448Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.417956914Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418149409Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418198673Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418272008Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418280959Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418317702Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418400308Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418430251Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418507866Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.4185447Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418631656Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418712042Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418758526Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418802879Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418866354Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418877005Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.418918248Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419037017Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419178558Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419289197Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41932898Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419404336Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419412016Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.41945853Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/istiod-openshift-gateway-94bb8fbfd-m2d5h.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419570948Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419584489Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.4195913Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.419628153Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435592654Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435658219Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435734165Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435757897Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/openshift-ai-inference-openshift-default-655f7c58b4-962b9.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435887067Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435899978Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435909218Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.435995885Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.43658668Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.436726681Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.436760594Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router-default-c7f6c5c4f-cjdpd.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.436884803Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.436892504Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.436898685Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.43696467Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.43710002Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437170666Z istio/namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437217339Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437255892Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437294005Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437347199Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437398303Z machine_config_ondisk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437408374Z machine_config_termination_logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437418665Z monitoring/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437425505Z monitoring/alertmanager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437458958Z monitoring/alertmanager/status.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437551875Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437624861Z monitoring/prometheus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437676255Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437760131Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.437822066Z monitoring/prometheus/rules.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.438516629Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.438562023Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.438609436Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439498395Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439548869Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439583902Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439673609Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439739484Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439830571Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439861583Z monitoring/prometheus/status/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.439901036Z monitoring/prometheus/status/config.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440365532Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440427097Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440510703Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440713789Z namespaces/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44072151Z namespaces/cert-manager-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44072568Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44072947Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440769853Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440833308Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440851109Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.440890232Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.441141412Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.441332046Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.441475668Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.441704805Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.441935893Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44202041Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-zngqj.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442184052Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442214565Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44228863Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442331814Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442384948Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44242019Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442496726Z namespaces/cert-manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442506677Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442520478Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442563701Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.442841833Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443043509Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443200201Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443403346Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443625433Z namespaces/default/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443675917Z namespaces/default/default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443743632Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443790156Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443860821Z namespaces/default/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.443908045Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444015524Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44409564Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444172776Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444218149Z namespaces/default/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444265293Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444311116Z namespaces/default/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44435363Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444426835Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44449711Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444536704Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44462189Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444709397Z namespaces/default/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44474501Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444861679Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.444972907Z namespaces/default/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445179473Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44527243Z namespaces/default/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445380049Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44552673Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445719835Z namespaces/default/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445797031Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445843864Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445923331Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.445998126Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446094354Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446145188Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446246435Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446342893Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446404357Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446458772Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446516826Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44657001Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446627345Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446634915Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446683759Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.446857512Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447065679Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44721136Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447417176Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447616091Z namespaces/default/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447655814Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447721669Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447772053Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447833858Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447841298Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447848089Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.447929095Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448039094Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448055745Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448080027Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448216807Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448306434Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448319255Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.44837609Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448499039Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448567684Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448623069Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448727047Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448797372Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448810193Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.448872708Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.449047481Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.449238406Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.449393968Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.449600304Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.449831372Z namespaces/kserve/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.449880526Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450014926Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45006903Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450155507Z namespaces/kserve/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45020127Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450306029Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450524995Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450727651Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450777925Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450867092Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.450913955Z namespaces/kserve/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451014863Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45110861Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451193367Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45124057Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451354259Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451410534Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451419824Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45149707Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451608159Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451728998Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45187941Z namespaces/kserve/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.451899281Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.452215336Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.452368377Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.452682192Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.452774979Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.453329712Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.453471042Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.453738123Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.453846171Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.453889585Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454033156Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45408249Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454173887Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454227441Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454320608Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454412555Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454484921Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454540515Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45461101Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454669855Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454766402Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454839138Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454893462Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.454987229Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455004371Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455054004Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455204996Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455395341Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455542032Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455746138Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.455992317Z namespaces/kserve/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456004568Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456067993Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/kserve-controller-manager-5876b4f96d-pgx27.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.45616735Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456184742Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456191082Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456260128Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456385477Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456478514Z namespaces/kserve/pods/kserve-controller-manager-5876b4f96d-pgx27/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456538279Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456604884Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/llmisvc-controller-manager-df5969cfc-jk8w2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456716333Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456729554Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456736254Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.456785228Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461258733Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.46134614Z namespaces/kserve/pods/llmisvc-controller-manager-df5969cfc-jk8w2/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461409115Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461466179Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/model-serving-api-fd65d7d6-wh8l4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461574747Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/server/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461582828Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/server/server/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461589279Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/server/server/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461657834Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/server/server/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461764892Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/server/server/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461848668Z namespaces/kserve/pods/model-serving-api-fd65d7d6-wh8l4/server/server/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461890492Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.461964308Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/odh-model-controller-84b54b7f97-4wtpk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.462083217Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.462093447Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.462097618Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.462177634Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483106278Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483159932Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-4wtpk/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483217696Z namespaces/kserve/pods/s3-init-hq5mf/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483287162Z namespaces/kserve/pods/s3-init-hq5mf/s3-init-hq5mf.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483383089Z namespaces/kserve/pods/s3-init-hq5mf/download-hf-model/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48339452Z namespaces/kserve/pods/s3-init-hq5mf/download-hf-model/download-hf-model/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483408711Z namespaces/kserve/pods/s3-init-hq5mf/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483436803Z namespaces/kserve/pods/s3-init-hq5mf/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483541701Z namespaces/kserve/pods/s3-init-hq5mf/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483636239Z namespaces/kserve/pods/s3-init-hq5mf/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483681712Z namespaces/kserve/pods/s3-init-hq5mf/s3-init/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483710694Z namespaces/kserve/pods/s3-init-hq5mf/s3-init/s3-init/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.483720875Z namespaces/kserve/pods/s3-init-hq5mf/s3-init/s3-init/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48378334Z namespaces/kserve/pods/s3-init-hq5mf/s3-init/s3-init/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484102335Z namespaces/kserve/pods/s3-init-hq5mf/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48417564Z namespaces/kserve/pods/s3-init-hq5mf/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484194411Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484265337Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs-64568bcd49-8dccw.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484351784Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484364485Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs/seaweedfs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484394127Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484444781Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484659538Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484737973Z namespaces/kserve/pods/seaweedfs-64568bcd49-8dccw/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484786387Z namespaces/kserve/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484831531Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.484970041Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485024615Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485140695Z namespaces/kuadrant-system/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485191728Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485292556Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48534211Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485391964Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485451299Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.485553156Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48586195Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486153903Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486233919Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486282903Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48637873Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486428164Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486526901Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486581936Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486628319Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486735598Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486784731Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486791502Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486853236Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.486964585Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.487081904Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.487181622Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.487231246Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.4872814Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.487476885Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.487616895Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.488054139Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.488153647Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.488628524Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.488779855Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489018774Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489144223Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489193357Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489326997Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489377621Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489457637Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489506141Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489608609Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489703507Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.48975345Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489822145Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489887301Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.489958676Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.490052003Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.490062394Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.490129879Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.490354037Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.490603776Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.490798181Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491057461Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491272097Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491324061Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-g2j52.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491655587Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491710281Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491816999Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.491912667Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492032216Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.49208819Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492144855Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492261424Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492314178Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492424146Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492591179Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492699998Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492804055Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492811056Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492854359Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino-7c77957575-z7vd9.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492972348Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.492986549Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.49299171Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.493040744Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496051216Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496115261Z namespaces/kuadrant-system/pods/authorino-7c77957575-z7vd9/authorino/authorino/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496175855Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496207328Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/authorino-operator-6d85f6564-xhvv8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496316376Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496323297Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496342408Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496409984Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496657992Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496741319Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-xhvv8/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496794923Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496866179Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/dns-operator-controller-manager-65b49595d7-qm9ts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496961636Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.496977417Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.49701595Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497065954Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497180603Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497260729Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-qm9ts/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.4972718Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497325034Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin-85fd4b8b9d-2ndhv.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.4974059Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497413481Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497420171Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497454424Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497552532Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497621937Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-2ndhv/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497645579Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497709404Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497836883Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497844804Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497848864Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.497896128Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.505802508Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.505879574Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-9btlm/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.505937538Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506076159Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador-limitador-69574b596d-9r2d7.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506200679Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506209429Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506214529Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506268514Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506379002Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506462599Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-9r2d7/limitador/limitador/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506514533Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506581428Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/limitador-operator-controller-manager-6f9f468797-g7cbp.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506692786Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506701457Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506706358Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.506763912Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507025222Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507108489Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-g7cbp/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507156952Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507212757Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.50726405Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507332806Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.50738613Z namespaces/kube-node-lease/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507395921Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507406112Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507465166Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-128-170.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507577615Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-129-157.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507674402Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-132-105.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507739927Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507748998Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.507824874Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508060862Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508280399Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508434801Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508652108Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508878235Z namespaces/kube-public/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508885215Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508891426Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.508955251Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.509132834Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.509342081Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.509506063Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510071717Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510090228Z namespaces/kube-system/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510095039Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.51011276Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510176975Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.51023851Z namespaces/kube-system/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510295704Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510434415Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510525282Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.5106346Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510692715Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510744929Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510807324Z namespaces/kube-system/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510871709Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.510981997Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511030391Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511099096Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511192053Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511254398Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511266979Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511327054Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-vn34j5kp26v4leinipumc6veqe.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511442793Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.51154072Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511647579Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511815371Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511901318Z namespaces/kube-system/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.511957852Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.512154987Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.512781526Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.513074729Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.513166616Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.513461908Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.513600319Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.513912293Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514054754Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514108188Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514193855Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514245459Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514312414Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514362498Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514467906Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514554953Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514619588Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514674242Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514732316Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.51478639Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514854396Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514867987Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.514959454Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.515123276Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.515307691Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.515460322Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.515670669Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.515893146Z namespaces/kube-system/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.515953291Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516019276Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer-dh44v.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516099152Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516106602Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516111313Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516151866Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516500623Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516584319Z namespaces/kube-system/pods/global-pull-secret-syncer-dh44v/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516633853Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516680456Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer-skqw5.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516812077Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516819897Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516824128Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.516862511Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517423294Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.51750436Z namespaces/kube-system/pods/global-pull-secret-syncer-skqw5/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517552284Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517599327Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer-wg7tt.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517688614Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517698385Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517702625Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.517775101Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518125388Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518207044Z namespaces/kube-system/pods/global-pull-secret-syncer-wg7tt/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518267639Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518320083Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent-4ztl5.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518432152Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518443343Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518447203Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.518503727Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519385395Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519468771Z namespaces/kube-system/pods/konnectivity-agent-4ztl5/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519540587Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519592351Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent-h2n7j.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519680368Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519687578Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519693809Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.519724251Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520671554Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520754971Z namespaces/kube-system/pods/konnectivity-agent-h2n7j/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520803914Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520850658Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent-x6tpp.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520968987Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520980368Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.520984348Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.521053314Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.521957623Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52205045Z namespaces/kube-system/pods/konnectivity-agent-x6tpp/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522121636Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52217445Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522272018Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/haproxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522284149Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522291259Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522348413Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522644827Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522726703Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-128-170.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522775437Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522855433Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52295408Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/haproxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522978182Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.522999304Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523046807Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523326219Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523410075Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52346353Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523516254Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52359943Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/haproxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523606891Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523610901Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523657234Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.523921325Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524024453Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-105.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524073167Z namespaces/kube-system/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524131471Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524178815Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524187955Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524265481Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52437685Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524451296Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52450088Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524576525Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524624809Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524691344Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524698405Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524706086Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.524786332Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.525019519Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.525233416Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.525385788Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.525593004Z namespaces/open-cluster-management-554d0026-27d2-42dc-9d76-4c718de14/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52579687Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52580393Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52580816Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.525848223Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.525958732Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526042258Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526077941Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526126145Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526306919Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526492593Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526656906Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.526883563Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527128742Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527181676Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527274493Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527325697Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527412574Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527461508Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527691365Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527783493Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527885Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.527935794Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528017391Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528069815Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528094587Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528200825Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528256369Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528306583Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528411901Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.528463105Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52853155Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.529471053Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.52957113Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.529674229Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.529770876Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.529868483Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53008249Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530272135Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530572608Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530621531Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530700128Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530747641Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530831608Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.530883712Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531003451Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531095888Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531168304Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531214977Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531305154Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531354928Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531431514Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531479898Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531558924Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531570055Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531600727Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53177202Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.531957975Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532131748Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532341384Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532590813Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532641417Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532708343Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532756216Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532872955Z namespaces/openshift-apiserver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.532922379Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533030777Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533079871Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533148966Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53319631Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533298968Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533391185Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533490913Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533540027Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533604601Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533674967Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533735522Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533833119Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.533906485Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53397018Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534080308Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534169135Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534225669Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534344569Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534441346Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534541584Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534633651Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.534733008Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53488626Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535080815Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535150751Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535202735Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535278121Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535327064Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53540576Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535455254Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535556792Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535649269Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535722205Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535770559Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535851325Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535900098Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535981705Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.535993646Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.536059251Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.536229324Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.536418978Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.53656738Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.536773006Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.536972651Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537023415Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537113032Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537162266Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537230101Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537236941Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537243342Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537305817Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537392084Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537403034Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537467629Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537641973Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.537828637Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538053684Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538270211Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538465426Z namespaces/openshift-authentication/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538473717Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538481558Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538554383Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.538725256Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539058952Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539209614Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539427621Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539621465Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539629776Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539634746Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539694111Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539775697Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539799409Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.539845833Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540028747Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540222192Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540373554Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54058393Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540786485Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540795266Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540800007Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540855881Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540912595Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.540923806Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541016973Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541093999Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54110704Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541165575Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541326577Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541514391Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541665393Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.541863698Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542091336Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542151051Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542232347Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542279641Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542364127Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542428612Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542518329Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542613926Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542713574Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542769588Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542832383Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542894218Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.542966043Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543062381Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543115435Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54317709Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543268657Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543332472Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543340992Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543406767Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543519146Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54356777Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543686529Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543777826Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543880854Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.543989012Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54409121Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544254813Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544378252Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544556586Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54461303Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544687676Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54473796Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544786474Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544850129Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.544954677Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545025192Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545110299Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545184554Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545229788Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545269601Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545301723Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545344717Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545377049Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.5453832Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545434324Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545602187Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54578109Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.545919161Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546196932Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546377697Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54642423Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546474464Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546481205Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546541239Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546611864Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546670359Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546741965Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546797719Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546913808Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.546989204Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.5470708Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547130705Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547214741Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547274046Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547417997Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547516234Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547615692Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547673046Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.5477238Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547798306Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.54784721Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.547962339Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548038514Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548096639Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548195077Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548272633Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548279423Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548321476Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548430955Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548530472Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548656222Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548771161Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548821955Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.548954275Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.549060983Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.549394739Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.549470485Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.549678651Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.549819072Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.549934381Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550012957Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550072761Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550121385Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55018803Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550345923Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550395536Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55044455Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550508445Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550604783Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55070104Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550749834Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550815239Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.550875843Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55095767Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.551044676Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.551054537Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.551123222Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.551283325Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55147435Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.551632182Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.551838307Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552068835Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552079086Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552142931Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/aws-ebs-csi-driver-node-kn9kq.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55225857Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-driver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55226562Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-driver/csi-driver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552269441Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-driver/csi-driver/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552328315Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552443864Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55252694Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552574474Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-liveness-probe/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552581395Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552587505Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55265069Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552749318Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.552836475Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553070322Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-node-driver-registrar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553080373Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553087154Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553161689Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553272628Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553340983Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-kn9kq/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553395507Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553435881Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/aws-ebs-csi-driver-node-zmpg6.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553526388Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-driver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553539529Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-driver/csi-driver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55355613Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-driver/csi-driver/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553612854Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553722953Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553788208Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553829151Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-liveness-probe/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553836281Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553841232Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553883895Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.553998824Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55407137Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554106582Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-node-driver-registrar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554113183Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554126334Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554177238Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554264565Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55432915Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zmpg6/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554397025Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554432098Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/aws-ebs-csi-driver-node-zq899.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554524674Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-driver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554541086Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-driver/csi-driver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554551977Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-driver/csi-driver/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55459411Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554683567Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554750962Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554803996Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-liveness-probe/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554811937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554823978Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55485395Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.554958328Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555055515Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555096979Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-node-driver-registrar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555105209Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55511181Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555161384Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555258661Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555325606Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-zq899/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555364279Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555404482Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555471718Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555479158Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55550637Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555583996Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555625299Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555665373Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555704336Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555778802Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555785722Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555789882Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555834236Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.5558956Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555903491Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.555932723Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556121798Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556135409Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556178942Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556338114Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556516848Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556651219Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.556858135Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557083682Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557125305Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557226543Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557253295Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557338672Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557369884Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557584781Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557664507Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557753604Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557785656Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.55784193Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557920927Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.557989842Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558036316Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558074368Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558158775Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558208709Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558241931Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558324748Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558372581Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558382382Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558431336Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558515072Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.558562606Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.559567034Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.559669501Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.559815133Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.559886548Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560118796Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560264727Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560406038Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56056377Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560610384Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560674609Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560738834Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560798148Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560847642Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560907927Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.560967262Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561059589Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561142205Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561191559Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561251503Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561316588Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561369913Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561437218Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561487551Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561519974Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561574418Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561617021Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561635533Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561701988Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.561876692Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.562170194Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.562315115Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.562543463Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56276355Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.562865968Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.562908851Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned-7jfvb.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563050112Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563060553Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned/tuned/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563066283Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned/tuned/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563112687Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned/tuned/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563230866Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563298061Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-7jfvb/tuned/tuned/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563367076Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563399829Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned-kwccx.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563505437Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563518138Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned/tuned/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563523359Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned/tuned/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563531519Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned/tuned/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563659699Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563724034Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-kwccx/tuned/tuned/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563739165Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563793009Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned-spd2m.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563882916Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563892427Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned/tuned/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563897357Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned/tuned/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.563913329Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned/tuned/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56406233Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564130055Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-spd2m/tuned/tuned/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564143116Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564204661Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564250715Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564270566Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564330081Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56470275Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564747273Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-128-170.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564847051Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-129-157.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564927917Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-132-105.ec2.internal.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.564996672Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565042906Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565136233Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565175036Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565258902Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565302786Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565367531Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565408684Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56549205Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565599719Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565701036Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56574389Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565800254Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565841767Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565884431Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.565983418Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566052154Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566092897Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566175833Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566249329Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566297393Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566478496Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566559113Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566694243Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.566771919Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56690904Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567052431Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567223774Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567279308Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567320421Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567388397Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56742907Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567503526Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567544569Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567624835Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567705461Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567758145Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567772736Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567838151Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.567887155Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568009185Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568027916Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568097351Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568145995Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568183938Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568191968Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568236312Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568387174Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568569688Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568712349Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.568924725Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569274732Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569294684Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569308885Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-56fcf86bf-sv5mc.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569421173Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-watch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569431664Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569440595Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569479248Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569578015Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569650101Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569695194Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569707375Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.569720536Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.56977265Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570510867Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570580233Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-56fcf86bf-sv5mc/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570636217Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570652628Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570730124Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570757636Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570871385Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.570909638Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571001825Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571020457Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571112734Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571132595Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571233043Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571335081Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57144981Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571494013Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571526736Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57157862Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571611442Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571695179Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571746763Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571794916Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571875563Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571922626Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571932297Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.571999012Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572084889Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572178506Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572199568Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572369341Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572478439Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572832897Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.572925704Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573099027Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57326816Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573441183Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57352878Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573577834Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57365849Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573736026Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573808502Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573844964Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573963874Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.573999066Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574120586Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574211363Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574290309Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574344023Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574417279Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574466592Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574591822Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574644746Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574749324Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574803238Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574914967Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574923788Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.574983442Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.575148175Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.575455119Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57560115Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.575808796Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576017982Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576028233Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576075286Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator-5d456fdbcd-9ztxj.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576164423Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576171944Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576175874Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576224238Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576327696Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576411112Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-5d456fdbcd-9ztxj/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576454096Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57650842Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576590156Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576598907Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576659292Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576731427Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57677246Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576846946Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.576892449Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577015589Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577094225Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57715868Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577209444Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57728874Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577330043Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577431821Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577524748Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577618365Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57767837Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577734004Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57780665Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.577865014Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578036087Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578112853Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578171428Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578266675Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.57832579Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578333401Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578385645Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578478192Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578512844Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578621533Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578704639Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578806937Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.578916346Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579017653Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579141173Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579312276Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579378861Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579424645Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579474469Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579511081Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579583037Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579662413Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579772612Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579870639Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579927653Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.579997749Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580052883Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580108978Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580174312Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580210435Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58026857Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580439093Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580626618Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580771879Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.580999916Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581199202Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581240675Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581325161Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581367524Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581510675Z namespaces/openshift-config-managed/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58156874Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581643796Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581684289Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581768055Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581817019Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.581918397Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582025375Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582118833Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582179157Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582233281Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582304277Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582362111Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582457748Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582532864Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582570457Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582686696Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582774373Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.582839478Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.586319756Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.586413293Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.586522782Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.586611779Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.586705886Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58688604Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58702103Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587084445Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587124598Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587244628Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587305392Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587378078Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587437923Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587486136Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587561692Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587599255Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587713324Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58779328Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.587868376Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58792651Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588041079Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588088143Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588161668Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58817424Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588248775Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588474272Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588679428Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.588822429Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589060428Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589276705Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589325358Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589417255Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589425166Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.5894786Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589549875Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589590118Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589663334Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589703837Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589840168Z namespaces/openshift-config-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589846988Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589852939Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.589906883Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.58999607Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.590007791Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.590076486Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.590235918Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.590420013Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.590563564Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.590769609Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591174531Z namespaces/openshift-config/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591220664Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591473404Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591517607Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591621485Z namespaces/openshift-config/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591665518Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591771527Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591863884Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.591977833Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592034007Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592089051Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592146086Z namespaces/openshift-config/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.59219979Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592313428Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592351982Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592416816Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592513464Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592559327Z namespaces/openshift-config/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592621362Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592746092Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592834339Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.592931816Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593049015Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593141452Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593310655Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593466447Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593536883Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593580156Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593668283Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593713876Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593783722Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593828635Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593906011Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.593966386Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594131549Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594220246Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594294001Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594340365Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.59440971Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594443723Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594526369Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.59453568Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594591644Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594750026Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.594964223Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595119455Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595332631Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595521456Z namespaces/openshift-config/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595568739Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595638275Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595683999Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595820159Z namespaces/openshift-console-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595876203Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.595989082Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596039186Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596104071Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596155485Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596256713Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596386243Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596504342Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596560416Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596601639Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596670484Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596707798Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596814446Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596859799Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.596925794Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597036983Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597093737Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597103048Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597150072Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597218537Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597269261Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597872587Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.597995057Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.598255857Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.598343744Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.598483044Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.598615945Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.598805999Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.598965921Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599031257Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599115383Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599166637Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599243183Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599291817Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599391414Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599486132Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599554507Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599609271Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599680917Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.59972867Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599803976Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.59985325Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.599923785Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.59998697Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.600048435Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.600080538Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.600146243Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.600300294Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.600484609Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60063707Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.600846466Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.601051402Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.601065203Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.601123538Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/console-operator-68b757fb67-ms4r4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.601326243Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/console-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.601338815Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/console-operator/console-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.601349985Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/console-operator/console-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60141507Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/console-operator/console-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602044119Z namespaces/openshift-console-operator/pods/console-operator-68b757fb67-ms4r4/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602107694Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602166858Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602226713Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602290708Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602358333Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602367694Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602377464Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602426888Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602591001Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602774265Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.602962009Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60322728Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.603437856Z namespaces/openshift-console/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60348507Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.603847228Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.603895072Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.603996009Z namespaces/openshift-console/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604043623Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604148361Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604315354Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604576224Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60464677Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604710184Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604790531Z namespaces/openshift-console/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604838754Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.604969344Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.605030909Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.605083963Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.605184311Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.605251216Z namespaces/openshift-console/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.605292339Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.605908307Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.606031126Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.606290656Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.606380133Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.606577888Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.606714169Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.606907744Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607004911Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607061726Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607146752Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607199746Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607287993Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607323166Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607424924Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607516871Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607587066Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60764095Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607717186Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60776914Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607845026Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60789727Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607987617Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.607999268Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.608051752Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.608206044Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.60840899Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.608563352Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.608758536Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.608982144Z namespaces/openshift-console/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.608993495Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609039548Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console-cb757cd9b-n89wz.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609138836Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609146046Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console/console/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609150037Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console/console/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609198641Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console/console/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609312379Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console/console/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609399066Z namespaces/openshift-console/pods/console-cb757cd9b-n89wz/console/console/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609438579Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609495393Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/downloads-685bfb5f64-j5m2p.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609593701Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/download-server/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609600641Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/download-server/download-server/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609604562Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/download-server/download-server/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.609651235Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/download-server/download-server/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610030845Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610111991Z namespaces/openshift-console/pods/downloads-685bfb5f64-j5m2p/download-server/download-server/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610150084Z namespaces/openshift-console/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610207598Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610286024Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610345829Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610460298Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610514602Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610586608Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610637051Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610712487Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.610768591Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61087464Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611070665Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611170362Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611236387Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611288211Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611363908Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6114038Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611508859Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611579274Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611630868Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611725815Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611822613Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.611861316Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612010367Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612109325Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612204612Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612297689Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612393077Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612529437Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612709091Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612781807Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612839071Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612914457Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.612988303Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613047167Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613102852Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613195099Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613290976Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613350721Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613406785Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613463779Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613516854Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613575988Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613587539Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613633743Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.613792165Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61399504Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614139211Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614344367Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614526581Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614565264Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61463763Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614675973Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614775801Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614836615Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614912511Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.614974086Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615068163Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615110136Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615216975Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615311212Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615403459Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615468994Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615530209Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615585223Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615643467Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615732004Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61580199Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615867235Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.615989504Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616049389Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616136696Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616175408Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616279796Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61632832Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61645235Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616544327Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616629463Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61670794Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616805397Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.616924286Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617110011Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617174576Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617215079Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617283514Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617321957Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617394073Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617437866Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617536033Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61762349Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617733969Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617778372Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617845607Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617888731Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617962176Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.617977277Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.618104047Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.618263709Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.618450274Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.618596055Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.618933141Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619165119Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6191832Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619274887Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619322691Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619417709Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619471683Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619665057Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619715792Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619806928Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.619855882Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.61995755Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620071379Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620158566Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62021954Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620261634Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62034392Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620387583Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620491841Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620565087Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620623211Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620716859Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620803255Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620848129Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.620986619Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621084557Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621192746Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621282322Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6213776Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62151423Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621703075Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621802062Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621854646Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621922532Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.621987937Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622052562Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622106246Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622204093Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622302771Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622350175Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62241528Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622473484Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622532279Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622601704Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622651778Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622723413Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622768587Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622846993Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622856754Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.622955871Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.623137445Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.623313309Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.623471781Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.623702419Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.623885663Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.623934197Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62410596Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.624162354Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.624265352Z namespaces/openshift-dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.624310106Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.624404003Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.624449686Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625195914Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625242007Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625417431Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625507458Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625606236Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625675771Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625733316Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625806751Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625871436Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.625984485Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626059141Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626101944Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626220353Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62630722Z namespaces/openshift-dns/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626364224Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626494884Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626599672Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626848602Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.626964871Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.627376982Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.627518383Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.627715768Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.627793744Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.627837258Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.627925585Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62799227Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628086947Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62812972Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628247219Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628350777Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628404071Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628465806Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628538572Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628572775Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628661201Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628700704Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6287765Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628788401Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.628818914Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629002438Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629194752Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629332693Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629539159Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629740774Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629752635Z namespaces/openshift-dns/pods/dns-default-99ggf/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.62980521Z namespaces/openshift-dns/pods/dns-default-99ggf/dns-default-99ggf.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629905307Z namespaces/openshift-dns/pods/dns-default-99ggf/dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629915588Z namespaces/openshift-dns/pods/dns-default-99ggf/dns/dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.629925149Z namespaces/openshift-dns/pods/dns-default-99ggf/dns/dns/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630000534Z namespaces/openshift-dns/pods/dns-default-99ggf/dns/dns/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630126954Z namespaces/openshift-dns/pods/dns-default-99ggf/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630208271Z namespaces/openshift-dns/pods/dns-default-99ggf/dns/dns/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630255404Z namespaces/openshift-dns/pods/dns-default-99ggf/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630269655Z namespaces/openshift-dns/pods/dns-default-99ggf/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630275756Z namespaces/openshift-dns/pods/dns-default-99ggf/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630350441Z namespaces/openshift-dns/pods/dns-default-99ggf/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63046458Z namespaces/openshift-dns/pods/dns-default-99ggf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630553917Z namespaces/openshift-dns/pods/dns-default-99ggf/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630595491Z namespaces/openshift-dns/pods/dns-default-fpvfk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630661196Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns-default-fpvfk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630755223Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630766864Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns/dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630777314Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns/dns/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63084877Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns/dns/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.630958699Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631061496Z namespaces/openshift-dns/pods/dns-default-fpvfk/dns/dns/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63110155Z namespaces/openshift-dns/pods/dns-default-fpvfk/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63111476Z namespaces/openshift-dns/pods/dns-default-fpvfk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631140033Z namespaces/openshift-dns/pods/dns-default-fpvfk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631209938Z namespaces/openshift-dns/pods/dns-default-fpvfk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631328977Z namespaces/openshift-dns/pods/dns-default-fpvfk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631418304Z namespaces/openshift-dns/pods/dns-default-fpvfk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631460127Z namespaces/openshift-dns/pods/dns-default-wrxt4/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631520972Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns-default-wrxt4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63162326Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6316348Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns/dns/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631643971Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns/dns/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631709386Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns/dns/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631831576Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.631916812Z namespaces/openshift-dns/pods/dns-default-wrxt4/dns/dns/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632072554Z namespaces/openshift-dns/pods/dns-default-wrxt4/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632089026Z namespaces/openshift-dns/pods/dns-default-wrxt4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632099456Z namespaces/openshift-dns/pods/dns-default-wrxt4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632155461Z namespaces/openshift-dns/pods/dns-default-wrxt4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632262989Z namespaces/openshift-dns/pods/dns-default-wrxt4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632363987Z namespaces/openshift-dns/pods/dns-default-wrxt4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632429772Z namespaces/openshift-dns/pods/node-resolver-4nx9g/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632487926Z namespaces/openshift-dns/pods/node-resolver-4nx9g/node-resolver-4nx9g.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632594974Z namespaces/openshift-dns/pods/node-resolver-4nx9g/dns-node-resolver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632607266Z namespaces/openshift-dns/pods/node-resolver-4nx9g/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632617086Z namespaces/openshift-dns/pods/node-resolver-4nx9g/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63266624Z namespaces/openshift-dns/pods/node-resolver-4nx9g/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632755557Z namespaces/openshift-dns/pods/node-resolver-4nx9g/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632821752Z namespaces/openshift-dns/pods/node-resolver-4nx9g/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632882147Z namespaces/openshift-dns/pods/node-resolver-pk5f2/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.632910229Z namespaces/openshift-dns/pods/node-resolver-pk5f2/node-resolver-pk5f2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633013317Z namespaces/openshift-dns/pods/node-resolver-pk5f2/dns-node-resolver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633023748Z namespaces/openshift-dns/pods/node-resolver-pk5f2/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633027828Z namespaces/openshift-dns/pods/node-resolver-pk5f2/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633064991Z namespaces/openshift-dns/pods/node-resolver-pk5f2/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633173989Z namespaces/openshift-dns/pods/node-resolver-pk5f2/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633247935Z namespaces/openshift-dns/pods/node-resolver-pk5f2/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633286158Z namespaces/openshift-dns/pods/node-resolver-z8fr5/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633346003Z namespaces/openshift-dns/pods/node-resolver-z8fr5/node-resolver-z8fr5.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633426229Z namespaces/openshift-dns/pods/node-resolver-z8fr5/dns-node-resolver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633434219Z namespaces/openshift-dns/pods/node-resolver-z8fr5/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63343809Z namespaces/openshift-dns/pods/node-resolver-z8fr5/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633494154Z namespaces/openshift-dns/pods/node-resolver-z8fr5/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633611493Z namespaces/openshift-dns/pods/node-resolver-z8fr5/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633690729Z namespaces/openshift-dns/pods/node-resolver-z8fr5/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633726882Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633790067Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63383538Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.633933848Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634036656Z namespaces/openshift-etcd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634082829Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634163606Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63421938Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634265663Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634332868Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634428296Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634521043Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634622691Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634676175Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63473349Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634808335Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63486342Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.634980538Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635066315Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635119099Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635220557Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635320855Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635369508Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635489698Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635585325Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635680552Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635769069Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.635867307Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636022429Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636193582Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636255677Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636310701Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636362195Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63642309Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636491095Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63655365Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636645247Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636744274Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636796429Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636856863Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636913308Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.636987833Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.637038827Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.637086831Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.637130654Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.637296907Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.637485682Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.637639774Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63785234Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638074597Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63811231Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638186676Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638227849Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638342178Z namespaces/openshift-host-network/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638387491Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638476198Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638519182Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638611678Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638655722Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6387608Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638869498Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.638986258Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639039042Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639096716Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639164391Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639217105Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639315753Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639383698Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639433162Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63953702Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639622147Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.63967466Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639789369Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.639883337Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64006136Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640163748Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640257625Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640417378Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640537867Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640607952Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640659656Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640740243Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640782896Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640861262Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640903085Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.640998542Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641039566Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641149934Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641243972Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641299036Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64135199Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641408394Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641462018Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641531374Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641545285Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641597479Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641764812Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.641967987Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642121609Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642317884Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64252629Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642569624Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642636599Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642676302Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642762689Z namespaces/openshift-image-registry/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642827123Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642905089Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.642969645Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64303697Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643081983Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643204993Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643349644Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643548819Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643596083Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643656938Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643728893Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643784227Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643898746Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.643957591Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.644041417Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.644135635Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.644165627Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.644173757Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.644246293Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64433419Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.644374623Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.645341508Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.645441245Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.645723657Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.645818414Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646049912Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646198754Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646400449Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646472145Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646526909Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646599974Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646653539Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646725044Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646776688Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.646892297Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647006676Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647083392Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647144356Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647234563Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647294058Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647431299Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647532626Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647596601Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647651966Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647714111Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647773125Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647830029Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647871083Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.647923947Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.648291585Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.648596399Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.648752311Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.648990519Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.649209896Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.649216666Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64926253Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/image-registry-766c776c56-ctjz7.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.649380249Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/registry/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.64938991Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/registry/registry/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6493938Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/registry/registry/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.649455505Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/registry/registry/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651301437Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651383573Z namespaces/openshift-image-registry/pods/image-registry-766c776c56-ctjz7/registry/registry/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651436147Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651488311Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca-2fd5t.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651572548Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651579398Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca/node-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651583259Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca/node-ca/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651630182Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca/node-ca/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651806336Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651889662Z namespaces/openshift-image-registry/pods/node-ca-2fd5t/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.651952117Z namespaces/openshift-image-registry/pods/node-ca-8wthl/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652026033Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca-8wthl.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65211391Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65212063Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca/node-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652126691Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca/node-ca/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652182265Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca/node-ca/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652351478Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652435635Z namespaces/openshift-image-registry/pods/node-ca-8wthl/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652486178Z namespaces/openshift-image-registry/pods/node-ca-skw5s/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652540372Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca-skw5s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652628039Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652637Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca/node-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65264096Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca/node-ca/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652703065Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca/node-ca/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652871798Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.652956704Z namespaces/openshift-image-registry/pods/node-ca-skw5s/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65302513Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653074174Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653145359Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653185362Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653254447Z namespaces/openshift-infra/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653261088Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653264888Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653312232Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653466294Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653649748Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.653803Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654107884Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654335401Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654383075Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654477212Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654534836Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654610802Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654651955Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654778325Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654880453Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.654997282Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655129762Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655184336Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655252052Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655299965Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655392132Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655465998Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655523753Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65561676Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655702206Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6557564Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65587572Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.655995889Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656154251Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656243308Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656418472Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656552742Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656748387Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656828003Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.656886788Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65704976Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657101684Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65717757Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657226754Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657328272Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657422149Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657495145Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657546698Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657616474Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657672338Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657741044Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657752354Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657813109Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.657986392Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.658169807Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.658313478Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.658524674Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.658961808Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.658975709Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659042794Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/ingress-canary-lgl65.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659133371Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/serve-healthcheck-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659143572Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659150822Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659209677Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659307764Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.65937717Z namespaces/openshift-ingress-canary/pods/ingress-canary-lgl65/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659428614Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659468277Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/ingress-canary-v9drg.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659543312Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/serve-healthcheck-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659553033Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659566974Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659622749Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659734217Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659813973Z namespaces/openshift-ingress-canary/pods/ingress-canary-v9drg/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659873068Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.659930672Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/ingress-canary-whkx9.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660049491Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/serve-healthcheck-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660068983Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660077434Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660135568Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660241016Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660322792Z namespaces/openshift-ingress-canary/pods/ingress-canary-whkx9/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660361535Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660428141Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660493106Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6605505Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660670509Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660728084Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660820001Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660865805Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.660962152Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661019036Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661118704Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661204941Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661311359Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661377064Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661443839Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661491943Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661542147Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661642265Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66171197Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661773824Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661866402Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.661961429Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.662029794Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663027071Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663125199Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663257779Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663348936Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663445884Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663590125Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66378654Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.663864706Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66392037Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664022358Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664083143Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664140017Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664147047Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664218973Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66430736Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664361504Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664459062Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664551789Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664625755Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664679459Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664757845Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664818239Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.664910136Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665038276Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665112662Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665165446Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665235871Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665252623Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665297736Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665389973Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665400244Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665461829Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665631432Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665814886Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.665957007Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666186695Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666368449Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666408032Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666457406Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666489498Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666612828Z namespaces/openshift-ingress/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666657051Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666863917Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666877718Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.666992057Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667015019Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667106886Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667351074Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66754602Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667585833Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667629076Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667702322Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667750475Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667832162Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667883506Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.667923189Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668028527Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66807777Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668095022Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668138875Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668229712Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668308258Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668416077Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66845447Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668864461Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.668961009Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.669214338Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.669310386Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.669572546Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.669689015Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.66988496Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.669993028Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.67001848Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670104187Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670118138Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670165702Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670249168Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.67027738Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670340645Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670386829Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670466895Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670543471Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670606076Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670646149Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670709844Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670749917Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670803731Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670862585Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6709197Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670931701Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.670993835Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.671170639Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.671359824Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.671496554Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.671772786Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.672006013Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.672016894Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.672055447Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/istiod-openshift-gateway-94bb8fbfd-m2d5h.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.672179857Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.672190328Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.672198888Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.67222287Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688058961Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688129767Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-m2d5h/discovery/discovery/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688185481Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688219694Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/openshift-ai-inference-openshift-default-655f7c58b4-962b9.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688356264Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688587622Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688601633Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.688644427Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689247803Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-962b9/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689390854Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689424017Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router-default-c7f6c5c4f-cjdpd.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689543406Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689551786Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689556737Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689603721Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689711359Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689777344Z namespaces/openshift-ingress/pods/router-default-c7f6c5c4f-cjdpd/router/router/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689815107Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689866181Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.689916735Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.68998435Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690086248Z namespaces/openshift-insights/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690141042Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690225638Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690275452Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690343478Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69038016Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69050468Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690606028Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690720676Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69076211Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690797753Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69089109Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.690925372Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691013459Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691052532Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691193313Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691247057Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69128992Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691370837Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691448463Z namespaces/openshift-insights/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.691487586Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.692399536Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.692518805Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.692812248Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.692993652Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693318927Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693417704Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693597468Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693671634Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693726888Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693805725Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693855238Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.693959146Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694050763Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694124479Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694214756Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694254989Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694326685Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694380209Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694445924Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694499568Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694586675Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694683322Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694758338Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694808652Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694905889Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.694969644Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695076092Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695117226Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695193432Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695220644Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695279798Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695295909Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695359754Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.695596153Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69582639Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696012385Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696231652Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696480971Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696491282Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696530895Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/insights-operator-5c847dcfc4-d5888.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696654904Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/insights-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696667635Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/insights-operator/insights-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696677926Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/insights-operator/insights-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.696735811Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697203827Z namespaces/openshift-insights/pods/insights-operator-5c847dcfc4-d5888/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697516321Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697568835Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/insights-runtime-extractor-7hkrk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697669972Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697678433Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/exporter/exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697683164Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/exporter/exporter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697739428Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/exporter/exporter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697840165Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697917322Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/exporter/exporter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697933423Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/extractor/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697964635Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/extractor/extractor/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.697973116Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/extractor/extractor/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.6980331Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/extractor/extractor/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698129358Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698197803Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/extractor/extractor/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698231376Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698239216Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698251627Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698310032Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69841591Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698497786Z namespaces/openshift-insights/pods/insights-runtime-extractor-7hkrk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698560741Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698619596Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/insights-runtime-extractor-sn8f2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698716493Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698725824Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/exporter/exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698734125Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/exporter/exporter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.69880214Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/exporter/exporter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698911478Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.698996595Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/exporter/exporter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699031268Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/extractor/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699046029Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/extractor/extractor/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699051799Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/extractor/extractor/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699110743Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/extractor/extractor/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699234953Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699303778Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/extractor/extractor/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699347432Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699357002Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699369303Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699399926Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699494513Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699563769Z namespaces/openshift-insights/pods/insights-runtime-extractor-sn8f2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699767294Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699808197Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/insights-runtime-extractor-xvqd8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699908755Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699918506Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/exporter/exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699924056Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/exporter/exporter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.699995672Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/exporter/exporter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700086169Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700168825Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/exporter/exporter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700185596Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/extractor/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700195497Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/extractor/extractor/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700204308Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/extractor/extractor/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700254942Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/extractor/extractor/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70036551Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700452047Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/extractor/extractor/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70049584Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700507171Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700513552Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700525192Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700648432Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700715517Z namespaces/openshift-insights/pods/insights-runtime-extractor-xvqd8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700762711Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700808214Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700866749Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.700919113Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70100549Z namespaces/openshift-keda/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701022371Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701028851Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701068245Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701175173Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701224877Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701234527Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701283321Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701348266Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7013969Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701507378Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701592675Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70165996Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701670451Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701719135Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.701889408Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.702216003Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.702416798Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.702614764Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.702759955Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.702934438Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.702998814Z namespaces/openshift-keda/operators.coreos.com/installplans/install-pplnc.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703100801Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703141365Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703194298Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703231211Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703290686Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703318528Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703425286Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70347401Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703543225Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70360116Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703672265Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703724509Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703820957Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.703912964Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704028853Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704087217Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704138811Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704200956Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70424984Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704348838Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704411112Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704467207Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704564954Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704648631Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704706545Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704825004Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.704917791Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705071633Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705177361Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705277589Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705436761Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7056835Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705756466Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705818271Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705891397Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.705957911Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706052599Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706097102Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706191049Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706260875Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706321279Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706340711Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706421757Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706466731Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706524265Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706536696Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70659194Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706754623Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.706965239Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707120811Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707335228Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707530643Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707581257Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707659073Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707713357Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707827546Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70788954Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.707959496Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708027181Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708103457Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708160531Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708258249Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708353086Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708450554Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708508958Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708550331Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708628307Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708672441Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708778559Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708847734Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.708901969Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709014637Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709113095Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709152178Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709281788Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709370054Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709481723Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709586741Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.709688689Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.70983304Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710042657Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71008863Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710160195Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710209949Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710268704Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710307287Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710378672Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7104726Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710565927Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710621271Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710689016Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710772653Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710834728Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.710969398Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711036273Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711073586Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711176404Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711221208Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711288893Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711298423Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711355808Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71151492Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711700765Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.711847706Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712079484Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712279539Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712318512Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712393718Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712438701Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712517637Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712562181Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712645507Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712690021Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712760836Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712804279Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.712916928Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713023146Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713117134Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713161837Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713226962Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713287947Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713347231Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713439479Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713501563Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713551447Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713651245Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71372417Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713776124Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.713897514Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714074547Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714163314Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71424236Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714316276Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714459207Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714634651Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714694155Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714729798Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714783552Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714840096Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714895001Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.714933984Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715053663Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715133509Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715175722Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715219656Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715268109Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715315133Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715361827Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715404Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715446973Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715456074Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715499037Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715643778Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715825803Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.715990115Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716330311Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716521136Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716546438Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716607183Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716643586Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716716751Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716754844Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716823749Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716850822Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.716907456Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71696436Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717065268Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717141984Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717213009Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717258733Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717295836Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71735472Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717387533Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717458318Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717514593Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717549525Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717634272Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717673025Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717680465Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717725259Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717788004Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717813486Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.717956027Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718045314Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718326505Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718396111Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718508759Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718593536Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718739317Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7187824Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718840345Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718890259Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.718911461Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719002417Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719062972Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719146199Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719216524Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719269108Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719287079Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719344404Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719376806Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.71942866Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719438141Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719466453Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719622325Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719796039Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.719959591Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720163827Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720345951Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720381444Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720425707Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720470081Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720532706Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720572429Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720630923Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720656475Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.72071888Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720745822Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720828458Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.720909215Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721017823Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721067117Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.72111355Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721159204Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721198287Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721274613Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721321646Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721358439Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721428624Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721487489Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721533663Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721664203Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721765061Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721859048Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.721957835Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722059433Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722165641Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722323973Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722373957Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.72241138Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722461214Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722502627Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722572813Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.722609406Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.72279592Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.72292395Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723006486Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723560189Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723613203Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723646255Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723694549Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723720351Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723772325Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.723921237Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724134883Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724276024Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724474409Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724667914Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724711468Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724762581Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724809745Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724888281Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.724929894Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725031832Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725066015Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725123919Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725165752Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725247379Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725319955Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725399881Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725449514Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725493328Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725555933Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725599316Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725685263Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725720575Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725752658Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725828394Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725900489Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.725972775Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726068992Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726143058Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726225544Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.72629693Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726384687Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726497455Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726655667Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726709852Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726759836Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726864394Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726875824Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726915508Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.726992693Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727083951Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727157846Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727189289Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727239403Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727282676Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727326339Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727366312Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727375383Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727419337Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727571998Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727749902Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.727896063Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728122041Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728312615Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728351498Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728397352Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728426274Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728534052Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728575976Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728663532Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728702145Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728753369Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728795472Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.728877449Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729040622Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729265109Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729312352Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729356046Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729401959Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729449643Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729527059Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729575143Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729615166Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729695222Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729729964Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729738795Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729785479Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729840213Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.729881426Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730008996Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730099523Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730279887Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730359833Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730486373Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730609703Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730789676Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730867822Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730886324Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.730956039Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731010683Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731064307Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731105141Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731186667Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731260593Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731304196Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731333838Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731398333Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731441476Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73148427Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731494241Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731533964Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.731694796Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73187493Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732036973Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732250129Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732436683Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732446354Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732479227Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/kube-storage-version-migrator-operator-846bf668f7-g6r9d.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732614527Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/kube-storage-version-migrator-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732624608Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732635459Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732672222Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732794441Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-846bf668f7-g6r9d/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732911Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.732960114Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733015138Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733054571Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733105755Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733152668Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733213163Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733244686Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73329763Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733334793Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733414769Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733520737Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733617234Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733662198Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733700941Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733750665Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733789437Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733875044Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733928088Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.733979022Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73407793Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734115643Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734155076Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734255653Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734330709Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734454039Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734565368Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734674826Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.734812996Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735013622Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735061026Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735104239Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735149433Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735195506Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73524129Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735283933Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735386221Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73549898Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735543583Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735561844Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735621609Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735687774Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735748219Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73576759Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.735789832Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.736217105Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73640987Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7365445Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.736741725Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.736925379Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73693584Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737008116Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator-d57d58cc9-kw85c.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737106363Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/graceful-termination/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737113504Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/graceful-termination/graceful-termination/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737119994Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737169298Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737259355Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73732698Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737355263Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737359613Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator/migrator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737363313Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator/migrator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737414517Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator/migrator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737522936Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73758628Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-d57d58cc9-kw85c/migrator/migrator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737610652Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737661506Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73771257Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737747143Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737805057Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73784298Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737910565Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.737955929Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738037575Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738086479Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738173026Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738314706Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738436076Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.73848579Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738527993Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738558455Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738606799Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738714457Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738761381Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738811715Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738904602Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738966537Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.738981248Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739019381Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739109528Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739169632Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739199155Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739319004Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739413201Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739744517Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.739823273Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74004843Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74017268Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740362864Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740425409Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740463112Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740530317Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74056393Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740608623Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740641616Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740737203Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740813119Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740849392Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740895256Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.740962161Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741037967Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74108025Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741173727Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741232162Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741243872Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741287726Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741459819Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74173139Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.741886512Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742134901Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742308265Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74250344Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742541572Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-29mbx.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74264046Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742678503Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742730437Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742791852Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742859247Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.742915051Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743017099Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74303037Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743072193Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/lws-controller-manager-599d46c4cb-m6gx7.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743171531Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743178941Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743185162Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743245327Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743374967Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743462783Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-m6gx7/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743509157Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743556951Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/lws-controller-manager-599d46c4cb-pkpkn.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743637247Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743649128Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/manager/manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743654968Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/manager/manager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743699312Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/manager/manager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.743968853Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744050499Z namespaces/openshift-lws-operator/pods/lws-controller-manager-599d46c4cb-pkpkn/manager/manager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744094642Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744136246Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator-fd8ccff4c-559pb.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744225222Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744234683Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744256385Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74432907Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744517425Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74457951Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-559pb/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744613662Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744666246Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744723281Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744774665Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744827389Z namespaces/openshift-machine-api/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744835219Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74483965Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.744882553Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745158464Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745346049Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745506371Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745728278Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745922213Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745931734Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745936354Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.745999709Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.746111448Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.746206075Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.746213396Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74627306Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.746439143Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.746626918Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.746775259Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747031289Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747231084Z namespaces/openshift-marketplace/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747238425Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747242315Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747288258Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747366735Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747374215Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747417659Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747529387Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747616984Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747724042Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.747787187Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74782907Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.748028475Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74822127Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.748369772Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.748583108Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.748806675Z namespaces/openshift-monitoring/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.748846539Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.748933385Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.74899875Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.749057985Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.749111219Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.749272482Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.749710635Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750191342Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750452882Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750502276Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750540969Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750590123Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750704542Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750783798Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750841372Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.750973883Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.751033887Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.751082471Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.754223493Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.754373805Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.755596139Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.755683756Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.756613988Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.756757439Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757259937Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757441081Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757490725Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757638707Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.75768701Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757722523Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757785368Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757887326Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.757996514Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758059739Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758100412Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758263995Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758300838Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758419277Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758438738Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758616432Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758668536Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758775984Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.758967309Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759068377Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759317986Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759454957Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759581686Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759672724Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759779582Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759872529Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.759962936Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760004359Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760095866Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760174862Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760259869Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760354086Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760435762Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760599535Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760699523Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760819452Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.760901798Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761086982Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76118368Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761255535Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761287978Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761344402Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761352683Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761409487Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761638405Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.761854482Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762042956Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762243462Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762462628Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762500922Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762601129Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76261067Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762672195Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762850309Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762860599Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76286542Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.762896722Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763045513Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763107868Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76312592Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76313107Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763142381Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763195395Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763310224Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763372239Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763409432Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763415212Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763419452Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763464616Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763558393Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76365001Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763696674Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763703524Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763711665Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763763309Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763865067Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.763933522Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764005767Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764022519Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764028499Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764058351Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764148318Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764213174Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764237985Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764244316Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764248146Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76429358Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764388557Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764451472Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764483524Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764497765Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764511957Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764545239Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764631146Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764700231Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764735344Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764756295Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator-64786d9cbb-msxxf.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764857593Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764868264Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764874825Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.764909007Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765652974Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765716309Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-64786d9cbb-msxxf/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765769134Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765810037Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics-7c9dbb8b7d-k6gsf.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765913254Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-main/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765923205Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765929046Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.765996761Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766090908Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766161454Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766189406Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-self/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766196527Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766209568Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766235209Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766336377Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766405352Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766441165Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766450916Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766459437Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76649891Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766681894Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766800743Z namespaces/openshift-monitoring/pods/kube-state-metrics-7c9dbb8b7d-k6gsf/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766837636Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766873219Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server-66f4b65746-xzst4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.766996738Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767008989Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server/metrics-server/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767013419Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server/metrics-server/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767042072Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767133649Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767198414Z namespaces/openshift-monitoring/pods/metrics-server-66f4b65746-xzst4/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767240487Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76727978Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin-d5dcff7cf-d6qmx.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767358626Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767367607Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767371587Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76740991Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767501667Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767571953Z namespaces/openshift-monitoring/pods/monitoring-plugin-d5dcff7cf-d6qmx/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767606125Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767645868Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter-blcb8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767766848Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/init-textfile/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767779799Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/init-textfile/init-textfile/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767789439Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/init-textfile/init-textfile/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767830242Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.767902858Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768007226Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76805479Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768068281Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768074481Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768128036Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768250335Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7683132Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768366354Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768377215Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter/node-exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768394966Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter/node-exporter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768414758Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768543908Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768612943Z namespaces/openshift-monitoring/pods/node-exporter-blcb8/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768663217Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768693139Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter-k9bfl.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768816849Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/init-textfile/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768825159Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/init-textfile/init-textfile/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768829119Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/init-textfile/init-textfile/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768874573Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.768956879Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769060867Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769104101Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769114532Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769120372Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769150374Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769235271Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769302836Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769336559Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769344529Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter/node-exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76935238Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter/node-exporter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769385032Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769492011Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769557126Z namespaces/openshift-monitoring/pods/node-exporter-k9bfl/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769600319Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769634001Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter-l47fk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769732999Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/init-textfile/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76974326Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/init-textfile/init-textfile/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.76974911Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/init-textfile/init-textfile/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769783413Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769854929Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769923204Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769964787Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769982108Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.769990229Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770033322Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77012887Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770193065Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770230567Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770238178Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter/node-exporter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770244409Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter/node-exporter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770283012Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77038751Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770457735Z namespaces/openshift-monitoring/pods/node-exporter-l47fk/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770500188Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770536621Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics-54cf96dfcf-4dvqp.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770707604Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-main/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770718195Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770724316Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770759088Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.770895519Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771046721Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771085144Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-self/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771091244Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771096844Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771107465Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771119286Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771195822Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771226284Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771230895Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771237055Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771277238Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771380706Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771451552Z namespaces/openshift-monitoring/pods/openshift-state-metrics-54cf96dfcf-4dvqp/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771495905Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.771581492Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772046967Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772061239Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772066269Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772090841Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77221302Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772270955Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772285506Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772291837Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772300497Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772362502Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772445778Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772510423Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772544076Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772550356Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772554047Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772589199Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772677046Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772741851Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772769883Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772774734Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772778544Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772817377Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.772908054Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773002871Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773025553Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773039514Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773048725Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773091748Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773190966Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773255271Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773290774Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773298114Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773303935Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773350078Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.773929533Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77402972Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774081875Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774089705Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774093645Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774136529Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774240717Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774292111Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774335844Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774384128Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator-567f7b8b7c-9vh9l.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774474625Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774504097Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774508447Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774514908Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774606595Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77467273Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774693762Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774708693Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774715363Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.774763007Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775070071Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775134416Z namespaces/openshift-monitoring/pods/prometheus-operator-567f7b8b7c-9vh9l/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775156827Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775212132Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook-d559d66d5-6xrpk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775288457Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775300258Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775308729Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775339741Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775431079Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775494344Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-d559d66d5-6xrpk/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775535547Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775568339Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client-5d86454dd9-87pl2.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775724241Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775734502Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775740512Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775776905Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775862772Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775927287Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.775992702Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/reload/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776004003Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/reload/reload/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776014173Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/reload/reload/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776072238Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/reload/reload/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776159575Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77622634Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/reload/reload/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776264623Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776270703Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client/telemeter-client/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776274404Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776304786Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776457078Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776524233Z namespaces/openshift-monitoring/pods/telemeter-client-5d86454dd9-87pl2/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776556765Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77661795Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-querier-85dd8f7bcf-p8vw4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776756231Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776763051Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776772762Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776814165Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.776906882Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77701042Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777117159Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-rules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777127919Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77713346Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777167482Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777255439Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777321594Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777357767Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-web/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777363037Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777366888Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777411501Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777498108Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777568523Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777597956Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777602636Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777606586Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77765244Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777740747Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777805582Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777843005Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/prom-label-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777850715Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777854675Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777890758Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.777997786Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778073482Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778089864Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-query/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778097574Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-query/thanos-query/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778103125Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-query/thanos-query/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778151808Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778278908Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778340723Z namespaces/openshift-monitoring/pods/thanos-querier-85dd8f7bcf-p8vw4/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778379596Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778405568Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778462622Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778500335Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778608533Z namespaces/openshift-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778651467Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778802828Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778863333Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.778906527Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77894995Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779193199Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779430587Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779505093Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779551846Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779587929Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.77973646Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779847179Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779908294Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.779964088Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780016342Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780057235Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780134261Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780171294Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780217718Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780295714Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780354628Z namespaces/openshift-multus/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780391641Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780536412Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.780618079Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.781105946Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.781227895Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.781860454Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.781960942Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782121314Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782178829Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782211841Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782313049Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782395645Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782481182Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78257665Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782645565Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782735222Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782825399Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.782912835Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783009643Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783049396Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78310113Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783139673Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783192487Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78323487Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783284764Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783317937Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783405793Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78348492Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783535113Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783572736Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783630731Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783681475Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783719507Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783764761Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783801874Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783822315Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.783865469Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784042623Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784227867Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784362167Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784573864Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784760508Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784767399Z namespaces/openshift-multus/pods/multus-4mwbb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784807182Z namespaces/openshift-multus/pods/multus-4mwbb/multus-4mwbb.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784898899Z namespaces/openshift-multus/pods/multus-4mwbb/kube-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784906779Z namespaces/openshift-multus/pods/multus-4mwbb/kube-multus/kube-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78491075Z namespaces/openshift-multus/pods/multus-4mwbb/kube-multus/kube-multus/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.784959893Z namespaces/openshift-multus/pods/multus-4mwbb/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.785860943Z namespaces/openshift-multus/pods/multus-4mwbb/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.785930868Z namespaces/openshift-multus/pods/multus-4mwbb/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786093931Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786208019Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/multus-additional-cni-plugins-2tnhw.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786273285Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/bond-cni-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786280935Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786284905Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786334489Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786413015Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786488481Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786505582Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786514343Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/cni-plugins/cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786521974Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786573258Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786680366Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786742911Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786804305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/egress-router-binary-copy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786833578Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786840638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786853049Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.786968358Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787074826Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787104999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787109849Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787113729Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787158303Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787228638Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787293243Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787322265Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/routeoverride-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787326816Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787330446Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78737552Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787498829Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787602807Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787652721Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni-bincopy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787660502Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787664832Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787724187Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787847906Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.787971266Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788015759Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788021529Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78802549Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788094275Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788208804Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788304271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-2tnhw/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788355695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78842036Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/multus-additional-cni-plugins-8jdfh.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788853063Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/bond-cni-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788860364Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788864194Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.788901097Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789016786Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789093602Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789123504Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789127845Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/cni-plugins/cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789131665Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789173558Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789257305Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78932201Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789363083Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/egress-router-binary-copy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789372963Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789378084Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789401726Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789488352Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789554397Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78958357Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7895907Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789594581Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789634434Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789702999Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789771464Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789798576Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/routeoverride-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789802867Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789806537Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.78984549Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.789935047Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790023924Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790061487Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni-bincopy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790076308Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790090179Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790119071Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790212708Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790274173Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790323357Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790335968Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790343338Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79036346Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790447466Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790524732Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-8jdfh/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790579857Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7906302Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/multus-additional-cni-plugins-9gtlt.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7907547Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/bond-cni-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790761591Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790765761Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790797523Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790906022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.790984428Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7910132Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79101764Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/cni-plugins/cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791022141Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791060044Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79114729Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791212495Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791252588Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/egress-router-binary-copy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791259709Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791263669Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791295522Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791391659Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791454694Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791492617Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791499858Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791503848Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791542831Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791648699Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791740726Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791767378Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/routeoverride-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791771799Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791775499Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791821812Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.791910599Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792010007Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792027948Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni-bincopy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792040009Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79204404Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792087143Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792171599Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792250085Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792300179Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79230983Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792315801Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792328751Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792427879Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792496695Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-9gtlt/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792524997Z namespaces/openshift-multus/pods/multus-b9jth/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79256717Z namespaces/openshift-multus/pods/multus-b9jth/multus-b9jth.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792653436Z namespaces/openshift-multus/pods/multus-b9jth/kube-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792660567Z namespaces/openshift-multus/pods/multus-b9jth/kube-multus/kube-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.792664497Z namespaces/openshift-multus/pods/multus-b9jth/kube-multus/kube-multus/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79270532Z namespaces/openshift-multus/pods/multus-b9jth/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793239142Z namespaces/openshift-multus/pods/multus-b9jth/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793330279Z namespaces/openshift-multus/pods/multus-b9jth/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793368812Z namespaces/openshift-multus/pods/multus-fxrwr/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793403794Z namespaces/openshift-multus/pods/multus-fxrwr/multus-fxrwr.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793488931Z namespaces/openshift-multus/pods/multus-fxrwr/kube-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793496221Z namespaces/openshift-multus/pods/multus-fxrwr/kube-multus/kube-multus/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793500202Z namespaces/openshift-multus/pods/multus-fxrwr/kube-multus/kube-multus/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.793549316Z namespaces/openshift-multus/pods/multus-fxrwr/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.794536842Z namespaces/openshift-multus/pods/multus-fxrwr/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.794593986Z namespaces/openshift-multus/pods/multus-fxrwr/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79463974Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.794658141Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon-5g56w.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795074373Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795086714Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795091095Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795127677Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795213574Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79528723Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795315652Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795322132Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795326073Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795373636Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79554856Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795616565Z namespaces/openshift-multus/pods/network-metrics-daemon-5g56w/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795646047Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795668919Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon-dsk4h.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795766107Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795775877Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795780018Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795800779Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795893366Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.795961452Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796019656Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796031657Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796035657Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796080981Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796268095Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79632937Z namespaces/openshift-multus/pods/network-metrics-daemon-dsk4h/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796377434Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796394735Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon-nvq9w.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796496693Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796509004Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796514614Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796524565Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796614332Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796697508Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7967197Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796725961Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796732011Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796771554Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.796902584Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797005142Z namespaces/openshift-multus/pods/network-metrics-daemon-nvq9w/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797028054Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79710906Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797162534Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797173445Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797184756Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797283804Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797331517Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79736394Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797455377Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.7974944Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797542273Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797591337Z namespaces/openshift-must-gather-d8w7l/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797600198Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797606349Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79762862Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.797832046Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798054813Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798217265Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798474005Z namespaces/openshift-must-gather-d8w7l/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79866857Z namespaces/openshift-network-console/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798701893Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79878684Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798855665Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798930931Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.798974194Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799075382Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.79918854Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799289668Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799333611Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799404127Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799490994Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799549478Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799607443Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799629304Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799720852Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799748163Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799799938Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799872693Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799925987Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.799983272Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800114312Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800245122Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800379552Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800459369Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800654833Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800839368Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.800963687Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801055775Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801137871Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801203456Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801234118Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801797342Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801895329Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80190323Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80190858Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801914171Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801975475Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801985606Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801990567Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.801995297Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.802026959Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80203261Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80204054Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.802076493Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.802114126Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.802124277Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.802314461Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.802983763Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803152686Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803399955Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803635763Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803652245Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803662765Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin-6bffdf9778-gcwjv.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803794496Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803806676Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803812777Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.803821168Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8039807Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804046395Z namespaces/openshift-network-console/pods/networking-console-plugin-6bffdf9778-gcwjv/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80411368Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804147713Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804219979Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804270972Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804313106Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80437332Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804414794Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804463957Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804507761Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804552474Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804670653Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804753849Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804854567Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804915762Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.804986307Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805054503Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805100196Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805162461Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805194123Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805251248Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805288691Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80540645Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805470925Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805525329Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805646658Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805698542Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805757067Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.805874746Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.806097583Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.806347102Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80644104Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.806690879Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.806800897Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.806958499Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807033055Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807076579Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807142674Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807198558Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807304756Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807369571Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807422995Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80748612Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807541375Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807596039Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807657604Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.80773819Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807824976Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807882661Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.807923644Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808011491Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808059295Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808118039Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808249789Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808330345Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808345557Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808364438Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808529741Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808735657Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.808875418Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809132917Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809754455Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809768946Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809780517Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/network-check-source-989f8b8dc-6fsgb.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809901557Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/check-endpoints/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809919678Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/check-endpoints/check-endpoints/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809926699Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.809970162Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810131914Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810197319Z namespaces/openshift-network-diagnostics/pods/network-check-source-989f8b8dc-6fsgb/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810224671Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810287386Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-9gj6d.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810366542Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-container/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810379854Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810384484Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810393955Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810484942Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810555297Z namespaces/openshift-network-diagnostics/pods/network-check-target-9gj6d/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81058895Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810609431Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-sjr8w.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810697028Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-container/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810706179Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810710239Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810734321Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810840579Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810913235Z namespaces/openshift-network-diagnostics/pods/network-check-target-sjr8w/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.810932756Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811006262Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-v5t9z.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811078897Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-container/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811087878Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811094249Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811129271Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811215748Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811282433Z namespaces/openshift-network-diagnostics/pods/network-check-target-v5t9z/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811325496Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811355489Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811407383Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811414453Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811462377Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811539763Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811592237Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81163032Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811710576Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81176516Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811818354Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811853717Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811904531Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.811981777Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81201953Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812084575Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812106836Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812200644Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81227932Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812350875Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812398379Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812438382Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812493566Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812510308Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812600445Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812647968Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812687321Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812768087Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812814221Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812822232Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812858934Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812911429Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.812969603Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813082092Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813180519Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813247555Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813336171Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813401796Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813553178Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813658976Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813714891Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813764534Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813809138Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813849421Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.813899315Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81396055Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814008503Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814051466Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814130552Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814206938Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814255242Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814291905Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814345719Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814388012Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814434626Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814441807Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814477489Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814623981Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814829626Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.814987519Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815194645Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815377629Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815416222Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815441214Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815450354Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815507949Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815547172Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815599156Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81564664Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815679972Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815731466Z namespaces/openshift-network-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815770539Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815838584Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815881658Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.815928271Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816075662Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816185431Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816262077Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816344623Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816387957Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81642899Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816487244Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816527817Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81656307Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816614904Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816695381Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816743094Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816770596Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816859693Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816896006Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816906197Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.816982003Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.817038597Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81707905Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.817218681Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.817292707Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81746288Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.817800726Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818005791Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818085938Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818211537Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818267692Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818310185Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818398342Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818457386Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81850417Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818552544Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818591647Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818642991Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818679263Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818731518Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818758789Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818862317Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.818953084Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81901892Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819065133Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819116017Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.81915934Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819213144Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819254698Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819296301Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819349785Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819377747Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819382398Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819433341Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819581863Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819773418Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.819907628Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820142916Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820328881Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820336421Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820364963Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter-2r6dl.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8204467Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82045372Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82045756Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820492253Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82058071Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820647405Z namespaces/openshift-network-operator/pods/iptables-alerter-2r6dl/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820688828Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8207168Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter-dmr4s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820792506Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820799437Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820803387Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82083968Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.820926867Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821022784Z namespaces/openshift-network-operator/pods/iptables-alerter-dmr4s/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821038645Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821093969Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter-kbfl4.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821165765Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821176396Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821182776Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821208548Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821295755Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821365541Z namespaces/openshift-network-operator/pods/iptables-alerter-kbfl4/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821403973Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821442146Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82149025Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821531103Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821583727Z namespaces/openshift-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821595298Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821601219Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82162425Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821781042Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.821989629Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82213394Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822321324Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822506318Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822549082Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822611546Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822647529Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822691133Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.822726875Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823072482Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823151938Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823234645Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823280388Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823332702Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823368275Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823410268Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823486394Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823537048Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823582121Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823660087Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823705601Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823749864Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823852192Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.823934348Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824053068Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824131184Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82421019Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824332539Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824556216Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824626142Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824666585Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82473729Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824768983Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824817797Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82486173Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.824966688Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825045334Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825093648Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825137921Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825183965Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825225608Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825292023Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825319806Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825402482Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825410242Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825447235Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825592327Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82577431Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.825913741Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826132328Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826334314Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826366576Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826428631Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826456233Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826522238Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82654835Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826603925Z namespaces/openshift-operators/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826610655Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826614625Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.826648988Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82679796Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827016016Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827157717Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827354612Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827542677Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82758394Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827649295Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827676167Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827747273Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827785156Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82783877Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.827855961Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828264122Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828341519Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828435476Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828460988Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828500111Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828645672Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828674404Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828731958Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828758881Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828848107Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.82887899Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.828929764Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829044013Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829066964Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829073405Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829119068Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829158651Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829220896Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829436943Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829502688Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829831473Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.829960193Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.830480773Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83056169Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.830732523Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.830779656Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.830816279Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.830924687Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831165716Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83121284Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831295286Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831331039Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831368722Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831439767Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83146886Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831587109Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8316052Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831657524Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831666955Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831707328Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831779713Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831825717Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.831922864Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832044534Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832100858Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832151302Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832208977Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832258921Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832351808Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832407022Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832468877Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832477417Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832519341Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832611618Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832674313Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832711945Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832751799Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83277396Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.832837185Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.833021589Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.833191102Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.833338444Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.833529128Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.833740905Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.833756376Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83380716Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-node-4jtwj.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834013526Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834033137Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834041618Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83405988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834166218Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834232883Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834260255Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834266835Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834270856Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834299548Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834418487Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834486562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834513214Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/nbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834519275Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/nbdb/nbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834523045Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/nbdb/nbdb/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834574399Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/nbdb/nbdb/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834678717Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834743262Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834780365Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/northd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834790896Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/northd/northd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834796736Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/northd/northd/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834833249Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/northd/northd/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.834964879Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835027584Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/northd/northd/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835058676Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-acl-logging/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835065577Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835069417Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83511195Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835193727Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835245241Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835254872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-controller/ovn-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835260742Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835291654Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835543724Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83561864Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835637681Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835644882Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835650512Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.835690495Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83744324Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.837474303Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83756259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/sbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83757177Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/sbdb/sbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83757586Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/sbdb/sbdb/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.837597322Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/sbdb/sbdb/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.837720422Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.837789027Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-4jtwj/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83782956Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838006354Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-node-v8sxk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838155225Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838162536Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838166476Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83821401Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83834774Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838418315Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838451688Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838458428Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838462299Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838516093Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838663174Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83873511Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838766662Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/nbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838771953Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/nbdb/nbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838775703Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/nbdb/nbdb/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838838988Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/nbdb/nbdb/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.838972808Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839052034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839089617Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/northd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839096308Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/northd/northd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839100098Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/northd/northd/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839147482Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/northd/northd/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839271311Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.83938195Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/northd/northd/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839420203Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-acl-logging/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839424363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839469197Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839477567Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839609287Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839619148Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839666882Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839675052Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-controller/ovn-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839679093Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.839704144Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84016821Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.840216124Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.840267478Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.840280529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.840285929Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84029825Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842175575Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84223624Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842280053Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/sbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842288584Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/sbdb/sbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842293774Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/sbdb/sbdb/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842303975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/sbdb/sbdb/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842424494Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84249067Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-v8sxk/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842507911Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842588477Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-node-vmx6b.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842744439Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8427532Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84275973Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842769511Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.842912422Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843002109Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843027421Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843035672Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843041222Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843084315Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843196104Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843265019Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843283131Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/nbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843290341Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/nbdb/nbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843297992Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/nbdb/nbdb/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843346945Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/nbdb/nbdb/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843440943Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843514738Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843550641Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/northd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843557872Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/northd/northd/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843562192Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/northd/northd/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843582804Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/northd/northd/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843711033Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843781309Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/northd/northd/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843820462Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-acl-logging/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843827342Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843831193Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843849794Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.843982674Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844043819Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84405139Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-controller/ovn-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84405537Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844083672Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844655966Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844711621Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844738313Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844745713Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844749764Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.844789567Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846528591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846593176Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846629909Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/sbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846636799Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/sbdb/sbdb/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846641079Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/sbdb/sbdb/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846680562Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/sbdb/sbdb/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84677348Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846859396Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-vmx6b/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846875097Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.846969025Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847005778Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847013418Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847053961Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847148598Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847287159Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847332703Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847375866Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847458802Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847540779Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847590193Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847636956Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84768205Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84768945Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84769354Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847718022Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847777987Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847785138Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.847826311Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.848109293Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84833499Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.848489282Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.848714789Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.848917435Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84898217Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849050405Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849085828Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849134152Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849185286Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849260021Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849385641Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.84950102Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849550644Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849599898Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849643551Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849691295Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849767691Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849821225Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849867788Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.849967966Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850028821Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850042652Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850056603Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850120948Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850180203Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850303782Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850401949Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850614886Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850692282Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850837853Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.850994315Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851176969Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851234964Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851290558Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851353193Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851390356Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851453641Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851486053Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851592541Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851675708Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851736563Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851782156Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851848201Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851893504Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.851950089Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852019844Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852072468Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852082769Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852113832Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852300586Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852466019Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852751871Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.852961507Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853165562Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853173273Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853199075Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/service-ca-operator-5d46d9ffd4-f2zvs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853289792Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/service-ca-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853297473Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853302043Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853332426Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853509399Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d46d9ffd4-f2zvs/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853663051Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853701804Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853736827Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853791261Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853836984Z namespaces/openshift-service-ca/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853884628Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.853951823Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854006227Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.85404479Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854087034Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8541659Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854267087Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854363765Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854411519Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854458272Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854500916Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854532548Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854622775Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854667938Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854709072Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854789238Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854843492Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854851283Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854894256Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.854968852Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855018165Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855140715Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855218171Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855346551Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855426697Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855539216Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855663905Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855836319Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855893043Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.855953368Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856086738Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856125081Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856167014Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856196136Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856286373Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856363549Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856409193Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856439315Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856512271Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856545813Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856601047Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856610998Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856649671Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.856803663Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857012469Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857171061Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857370257Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857551361Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857560081Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857574722Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-758f599c5-q67lk.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8576648Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.85767552Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857681481Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.857705673Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858048929Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858112834Z namespaces/openshift-service-ca/pods/service-ca-758f599c5-q67lk/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858155367Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858182589Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858236374Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858278837Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858325731Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858370924Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858429548Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858468802Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858515825Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858546837Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858634594Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.85871322Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858791676Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.85884132Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858888684Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.858935338Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859000672Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859079239Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859127552Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859158755Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859247712Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859294105Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859332338Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859431006Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859508022Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859587448Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859668214Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859743Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.859864089Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860053924Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860117789Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860162442Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860204655Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860248699Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860295292Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860337226Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860417802Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860497098Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860548022Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860584905Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860632338Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860673361Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860720615Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860727766Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.860766289Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86091573Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861119366Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861283179Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861487804Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861671549Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861719002Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861759705Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861801439Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861862143Z namespaces/openshift/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861909827Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.861990413Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862043547Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862096371Z namespaces/openshift/apps/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862133014Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862219601Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862296667Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862376413Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862445168Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862493972Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862538485Z namespaces/openshift/batch/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862575258Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862677546Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86272521Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862767533Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862866721Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862911164Z namespaces/openshift/core/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.862953697Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863086567Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863164133Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86324596Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863322556Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863421063Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863555944Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863724087Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863777791Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863820884Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.863868448Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86389926Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86532547Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.865371514Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.865467601Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.865546927Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.865693028Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.865804157Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.865888494Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866088579Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866251542Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866376181Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866466958Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866562556Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866688835Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866793224Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.866924314Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867039632Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867125129Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867232537Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867346376Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867463385Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867555682Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86765711Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867759458Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867850965Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.867954303Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.868086653Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.868188151Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.868391397Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.868502365Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.868612794Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.868716682Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86882208Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86896116Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869054958Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869160796Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869249303Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869372432Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869515563Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.86960431Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869742921Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869845099Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.869960237Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870111029Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870201796Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870295183Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870424873Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870560774Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870665132Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870778451Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870885979Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.870998978Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871091315Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871209604Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871335424Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871475154Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871598364Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871720013Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871832742Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.871928579Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872138325Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872217922Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872264585Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872344171Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872425018Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872483782Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872531126Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872577649Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872621143Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872672177Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872679057Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872734411Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.872898234Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.20.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873115701Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873252011Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873445856Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873777722Z namespaces/openshift/policy/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873794033Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873846777Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873891971Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873929594Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.873935944Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874003089Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874113218Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874232067Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874337475Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874446674Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874608206Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874715834Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874839804Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.874954333Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875128886Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875232974Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875350453Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875476713Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875615054Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875727132Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875836231Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.875964781Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.8760856Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876187738Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876294956Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876399134Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876499562Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.87660222Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876709388Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876810556Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.876913794Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877035463Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877137581Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.87725331Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877361868Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877458836Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877560514Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877661061Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.877782541Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.87789717Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.87803594Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878132948Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878233146Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878343444Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878473764Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878565051Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.87868008Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878790598Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.878931729Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879078441Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879206581Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879350292Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879478662Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879603361Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.87972022Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879807217Z network_logs/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879838899Z network_logs/cluster_scale [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.879928846Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.880026204Z network_logs/multi-networkpolicy [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.88010327Z network_logs/net-attach-def [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.880184896Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.880333588Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.880420654Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.883236111Z nodes/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.883256523Z nodes/debug [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.883358831Z nodes/ip-10-0-128-170.ec2.internal/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.883388343Z nodes/ip-10-0-128-170.ec2.internal/cpu_affinities.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.883750651Z nodes/ip-10-0-128-170.ec2.internal/dmesg [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.88413095Z nodes/ip-10-0-128-170.ec2.internal/ethtool_channels [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.884209386Z nodes/ip-10-0-128-170.ec2.internal/ethtool_features [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.884296523Z nodes/ip-10-0-128-170.ec2.internal/ip-10-0-128-170.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.884656051Z nodes/ip-10-0-128-170.ec2.internal/irq_affinities.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.884803232Z nodes/ip-10-0-128-170.ec2.internal/lscpu [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.884930782Z nodes/ip-10-0-128-170.ec2.internal/lspci [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.885046781Z nodes/ip-10-0-128-170.ec2.internal/podresources.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.885152019Z nodes/ip-10-0-128-170.ec2.internal/pods_info.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.885278029Z nodes/ip-10-0-128-170.ec2.internal/proc_cmdline [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.885360755Z nodes/ip-10-0-128-170.ec2.internal/sysinfo.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.887177865Z nodes/ip-10-0-128-170.ec2.internal/sysinfo.tgz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.887318696Z nodes/ip-10-0-129-157.ec2.internal/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.88737133Z nodes/ip-10-0-129-157.ec2.internal/cpu_affinities.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.887613249Z nodes/ip-10-0-129-157.ec2.internal/dmesg [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.887871959Z nodes/ip-10-0-129-157.ec2.internal/ethtool_channels [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.887965336Z nodes/ip-10-0-129-157.ec2.internal/ethtool_features [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.888118608Z nodes/ip-10-0-129-157.ec2.internal/ip-10-0-129-157.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.888638008Z nodes/ip-10-0-129-157.ec2.internal/irq_affinities.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.888783859Z nodes/ip-10-0-129-157.ec2.internal/lscpu [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.888906618Z nodes/ip-10-0-129-157.ec2.internal/lspci [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.889015837Z nodes/ip-10-0-129-157.ec2.internal/podresources.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.889118855Z nodes/ip-10-0-129-157.ec2.internal/pods_info.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.889225123Z nodes/ip-10-0-129-157.ec2.internal/proc_cmdline [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.889306499Z nodes/ip-10-0-129-157.ec2.internal/sysinfo.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.891103098Z nodes/ip-10-0-129-157.ec2.internal/sysinfo.tgz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.891271741Z nodes/ip-10-0-132-105.ec2.internal/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.891305434Z nodes/ip-10-0-132-105.ec2.internal/cpu_affinities.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.891603737Z nodes/ip-10-0-132-105.ec2.internal/dmesg [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.891865847Z nodes/ip-10-0-132-105.ec2.internal/ethtool_channels [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.891954624Z nodes/ip-10-0-132-105.ec2.internal/ethtool_features [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.892049291Z nodes/ip-10-0-132-105.ec2.internal/ip-10-0-132-105.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.892534688Z nodes/ip-10-0-132-105.ec2.internal/irq_affinities.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.892675649Z nodes/ip-10-0-132-105.ec2.internal/lscpu [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.892798409Z nodes/ip-10-0-132-105.ec2.internal/lspci [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.892920418Z nodes/ip-10-0-132-105.ec2.internal/podresources.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.893048118Z nodes/ip-10-0-132-105.ec2.internal/pods_info.json [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.893166437Z nodes/ip-10-0-132-105.ec2.internal/proc_cmdline [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.893248333Z nodes/ip-10-0-132-105.ec2.internal/sysinfo.log [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.895033211Z nodes/ip-10-0-132-105.ec2.internal/sysinfo.tgz [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.895203314Z pod_network_connectivity_check/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.895252078Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.895345445Z static-pods/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.895355616Z static-pods/kube-apiserver/ [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.904196797Z [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.904214209Z sent 71,411 bytes received 8,648,384 bytes 5,813,196.67 bytes/sec [must-gather] [must-gather-78xgd] OUT 2026-07-09T11:00:17.904222159Z total size is 99,031,198 speedup is 11.36 [must-gather] [must-gather ] OUT 2026-07-09T11:00:18.202550473Z namespace/openshift-must-gather-d8w7l deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: 99737322-a04c-4b9b-8075-44dafcd2cd09 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.23" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-k8ndt [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-k8ndt-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-k8ndt [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 16212098 Jul 9 11:02 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-k8ndt/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 1538177] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-k8ndt/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] c9d3d99..1538177 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-09T11:02:34.129Z","type":3}]