task e2e-llm-inference-service has failed: "step-fail-if-needed" exited with code 1: Error [get-kubeconfig] Found kubeconfig secret: cluster-7xrph-admin-kubeconfig [get-kubeconfig] Wrote kubeconfig to /credentials/cluster-7xrph-kubeconfig [get-kubeconfig] Found admin password secret: cluster-7xrph-admin-password [get-kubeconfig] Retrieved username [get-kubeconfig] Wrote password to /credentials/cluster-7xrph-password [get-kubeconfig] API Server URL: https://a598157d2c94f4659a8dcc831855073a-75975bfcf6c3c9ed.elb.us-east-1.amazonaws.com:6443 [get-kubeconfig] Console URL: https://console-openshift-console.apps.b0136107-d2e9-40aa-b868-c21d27bac5a2.prod.konfluxeaas.com [clone-repo] odh-gateway-2gi-memory [clone-repo] https://github.com/jlost/kserve [clone-repo] Cloning into '/workspace/source'... [e2e-llm-inference-service] + bash [e2e-llm-inference-service] + STATUS_FILE=/test-status/deploy-and-e2e-status [e2e-llm-inference-service] + echo failed [e2e-llm-inference-service] + COMPONENT_NAME=kserve-agent-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-agent-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183 [e2e-llm-inference-service] + KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183 [e2e-llm-inference-service] + COMPONENT_NAME=kserve-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:4027d5c68e4b38dbdf70fa5c894b50b863d7857990c7284124c7a020c312a97c [e2e-llm-inference-service] + KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:4027d5c68e4b38dbdf70fa5c894b50b863d7857990c7284124c7a020c312a97c [e2e-llm-inference-service] + COMPONENT_NAME=kserve-router-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-router-ci '.[$component_name].image' [e2e-llm-inference-service] + export KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:78561fce36825084e3c48d0137c513c345701cffde73e206abb14ec219f8181b [e2e-llm-inference-service] + KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:78561fce36825084e3c48d0137c513c345701cffde73e206abb14ec219f8181b [e2e-llm-inference-service] + COMPONENT_NAME=kserve-storage-initializer-ci [e2e-llm-inference-service] ++ jq -r --arg component_name kserve-storage-initializer-ci '.[$component_name].image' [e2e-llm-inference-service] + export STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb [e2e-llm-inference-service] + STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb [e2e-llm-inference-service] + COMPONENT_NAME=odh-kserve-llmisvc-controller-ci [e2e-llm-inference-service] ++ jq -r --arg component_name odh-kserve-llmisvc-controller-ci '.[$component_name].image' [e2e-llm-inference-service] + export LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:e069958436927d1ba06d07865aae89800ab14a8c639ffa5751e1c39f1132cc2d [e2e-llm-inference-service] + LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:e069958436927d1ba06d07865aae89800ab14a8c639ffa5751e1c39f1132cc2d [e2e-llm-inference-service] + ./test/scripts/openshift-ci/run-e2e-tests.sh 'llminferenceservice and cluster_cpu and not autoscaling and not tracing' 2 llm-d [e2e-llm-inference-service] Installing on cluster [e2e-llm-inference-service] Using namespace: kserve for KServe components [e2e-llm-inference-service] SKLEARN_IMAGE=quay.io/opendatahub/sklearn-serving-runtime:odh-pr-1683 [e2e-llm-inference-service] OPT_125M_MODEL_URI=s3://example-models/facebook/opt-125m [e2e-llm-inference-service] ERROR_404_ISVC_IMAGE=quay.io/opendatahub/error-404-isvc:odh-pr-1683 [e2e-llm-inference-service] SUCCESS_200_ISVC_IMAGE=quay.io/opendatahub/success-200-isvc:odh-pr-1683 [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed Kustomize v5.8.1 to /workspace/source/bin/kustomize [e2e-llm-inference-service] v5.8.1 [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] [INFO] Installing yq v4.52.1 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed yq v4.52.1 to /workspace/source/bin/yq [e2e-llm-inference-service] yq (https://github.com/mikefarah/yq/) version v4.52.1 [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Installing KServe Python SDK ... [e2e-llm-inference-service] [INFO] Installing uv 0.7.8 for linux/amd64... [e2e-llm-inference-service] [SUCCESS] Successfully installed uv 0.7.8 to /workspace/source/bin/uv [e2e-llm-inference-service] warning: Failed to read project metadata (No `pyproject.toml` found in current directory or any parent directory). Running `uv self version` for compatibility. This fallback will be removed in the future; pass `--preview` to force an error. [e2e-llm-inference-service] uv 0.7.8 [e2e-llm-inference-service] Creating virtual environment... [e2e-llm-inference-service] warning: virtualenv's `--clear` has no effect (uv always clears the virtual environment) [e2e-llm-inference-service] Using CPython 3.9.25 interpreter at: /usr/bin/python3 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 [e2e-llm-inference-service] Creating virtual environment at: .venv [e2e-llm-inference-service] Resolved 266 packages in 1ms [e2e-llm-inference-service] Building kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading aiohttp (1.7MiB) [e2e-llm-inference-service] Downloading pydantic-core (2.0MiB) [e2e-llm-inference-service] Downloading pyarrow (40.1MiB) [e2e-llm-inference-service] Downloading kubernetes (1.9MiB) [e2e-llm-inference-service] Downloading cryptography (4.3MiB) [e2e-llm-inference-service] Downloading portforward (3.9MiB) [e2e-llm-inference-service] Downloading black (1.6MiB) [e2e-llm-inference-service] Downloading pandas (12.5MiB) [e2e-llm-inference-service] Downloading numpy (15.7MiB) [e2e-llm-inference-service] Downloading setuptools (1.2MiB) [e2e-llm-inference-service] Downloading grpcio-tools (2.5MiB) [e2e-llm-inference-service] Downloading uvloop (3.8MiB) [e2e-llm-inference-service] Downloading mypy (17.2MiB) [e2e-llm-inference-service] Downloading botocore (12.9MiB) [e2e-llm-inference-service] Downloading grpcio (6.4MiB) [e2e-llm-inference-service] Building timeout-sampler==1.0.3 [e2e-llm-inference-service] Building python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading aiohttp [e2e-llm-inference-service] Downloading pydantic-core [e2e-llm-inference-service] Downloading black [e2e-llm-inference-service] Downloading grpcio-tools [e2e-llm-inference-service] Downloading setuptools [e2e-llm-inference-service] Downloading portforward [e2e-llm-inference-service] Downloading uvloop [e2e-llm-inference-service] Downloading cryptography [e2e-llm-inference-service] Built python-simple-logger==2.0.19 [e2e-llm-inference-service] Downloading grpcio [e2e-llm-inference-service] Downloading kubernetes [e2e-llm-inference-service] Built timeout-sampler==1.0.3 [e2e-llm-inference-service] Downloading numpy [e2e-llm-inference-service] Built kserve @ file:///workspace/source/python/kserve [e2e-llm-inference-service] Downloading pandas [e2e-llm-inference-service] Downloading botocore [e2e-llm-inference-service] Downloading mypy [e2e-llm-inference-service] Downloading pyarrow [e2e-llm-inference-service] Prepared 101 packages in 1.83s [e2e-llm-inference-service] warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. [e2e-llm-inference-service] If the cache and target directories are on different filesystems, hardlinking may not be supported. [e2e-llm-inference-service] If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. [e2e-llm-inference-service] Installed 101 packages in 273ms [e2e-llm-inference-service] + aiohappyeyeballs==2.6.1 [e2e-llm-inference-service] + aiohttp==3.13.3 [e2e-llm-inference-service] + aiosignal==1.4.0 [e2e-llm-inference-service] + annotated-doc==0.0.4 [e2e-llm-inference-service] + annotated-types==0.7.0 [e2e-llm-inference-service] + anyio==4.9.0 [e2e-llm-inference-service] + attrs==25.3.0 [e2e-llm-inference-service] + avro==1.12.0 [e2e-llm-inference-service] + black==24.3.0 [e2e-llm-inference-service] + boto3==1.37.35 [e2e-llm-inference-service] + botocore==1.37.35 [e2e-llm-inference-service] + cachetools==5.5.2 [e2e-llm-inference-service] + certifi==2025.1.31 [e2e-llm-inference-service] + cffi==2.0.0 [e2e-llm-inference-service] + charset-normalizer==3.4.1 [e2e-llm-inference-service] + click==8.1.8 [e2e-llm-inference-service] + cloudevents==1.11.0 [e2e-llm-inference-service] + colorama==0.4.6 [e2e-llm-inference-service] + colorlog==6.10.1 [e2e-llm-inference-service] + coverage==7.8.0 [e2e-llm-inference-service] + cryptography==46.0.5 [e2e-llm-inference-service] + deprecation==2.1.0 [e2e-llm-inference-service] + durationpy==0.9 [e2e-llm-inference-service] + execnet==2.1.1 [e2e-llm-inference-service] + fastapi==0.136.3 [e2e-llm-inference-service] + frozenlist==1.5.0 [e2e-llm-inference-service] + google-auth==2.39.0 [e2e-llm-inference-service] + grpc-interceptor==0.15.4 [e2e-llm-inference-service] + grpcio==1.78.1 [e2e-llm-inference-service] + grpcio-testing==1.78.1 [e2e-llm-inference-service] + grpcio-tools==1.78.1 [e2e-llm-inference-service] + h11==0.16.0 [e2e-llm-inference-service] + httpcore==1.0.9 [e2e-llm-inference-service] + httptools==0.6.4 [e2e-llm-inference-service] + httpx==0.27.2 [e2e-llm-inference-service] + httpx-retries==0.4.5 [e2e-llm-inference-service] + idna==3.10 [e2e-llm-inference-service] + iniconfig==2.1.0 [e2e-llm-inference-service] + jinja2==3.1.6 [e2e-llm-inference-service] + jmespath==1.0.1 [e2e-llm-inference-service] + kserve==0.19.0 (from file:///workspace/source/python/kserve) [e2e-llm-inference-service] + kubernetes==32.0.1 [e2e-llm-inference-service] + markupsafe==3.0.2 [e2e-llm-inference-service] + multidict==6.4.3 [e2e-llm-inference-service] + mypy==0.991 [e2e-llm-inference-service] + mypy-extensions==1.0.0 [e2e-llm-inference-service] + numpy==2.2.4 [e2e-llm-inference-service] + oauthlib==3.2.2 [e2e-llm-inference-service] + orjson==3.10.16 [e2e-llm-inference-service] + packaging==24.2 [e2e-llm-inference-service] + pandas==2.2.3 [e2e-llm-inference-service] + pathspec==0.12.1 [e2e-llm-inference-service] + platformdirs==4.3.7 [e2e-llm-inference-service] + pluggy==1.5.0 [e2e-llm-inference-service] + portforward==0.7.1 [e2e-llm-inference-service] + prometheus-client==0.21.1 [e2e-llm-inference-service] + propcache==0.3.1 [e2e-llm-inference-service] + protobuf==6.33.5 [e2e-llm-inference-service] + psutil==5.9.8 [e2e-llm-inference-service] + pyarrow==19.0.1 [e2e-llm-inference-service] + pyasn1==0.6.3 [e2e-llm-inference-service] + pyasn1-modules==0.4.2 [e2e-llm-inference-service] + pycparser==2.22 [e2e-llm-inference-service] + pydantic==2.12.4 [e2e-llm-inference-service] + pydantic-core==2.41.5 [e2e-llm-inference-service] + pyjwt==2.12.1 [e2e-llm-inference-service] + pytest==7.4.4 [e2e-llm-inference-service] + pytest-asyncio==0.23.8 [e2e-llm-inference-service] + pytest-cov==5.0.0 [e2e-llm-inference-service] + pytest-httpx==0.30.0 [e2e-llm-inference-service] + pytest-json-report==1.5.0 [e2e-llm-inference-service] + pytest-metadata==3.1.1 [e2e-llm-inference-service] + pytest-xdist==3.6.1 [e2e-llm-inference-service] + python-dateutil==2.9.0.post0 [e2e-llm-inference-service] + python-dotenv==1.1.0 [e2e-llm-inference-service] + python-multipart==0.0.22 [e2e-llm-inference-service] + python-simple-logger==2.0.19 [e2e-llm-inference-service] + pytz==2025.2 [e2e-llm-inference-service] + pyyaml==6.0.2 [e2e-llm-inference-service] + requests==2.32.3 [e2e-llm-inference-service] + requests-oauthlib==2.0.0 [e2e-llm-inference-service] + rsa==4.9.1 [e2e-llm-inference-service] + s3transfer==0.11.4 [e2e-llm-inference-service] + setuptools==78.1.0 [e2e-llm-inference-service] + six==1.17.0 [e2e-llm-inference-service] + sniffio==1.3.1 [e2e-llm-inference-service] + starlette==1.2.1 [e2e-llm-inference-service] + tabulate==0.9.0 [e2e-llm-inference-service] + timeout-sampler==1.0.3 [e2e-llm-inference-service] + timing-asgi==0.3.1 [e2e-llm-inference-service] + tomlkit==0.13.2 [e2e-llm-inference-service] + typing-extensions==4.15.0 [e2e-llm-inference-service] + typing-inspection==0.4.2 [e2e-llm-inference-service] + tzdata==2025.2 [e2e-llm-inference-service] + urllib3==2.6.2 [e2e-llm-inference-service] + uvicorn==0.34.1 [e2e-llm-inference-service] + uvloop==0.21.0 [e2e-llm-inference-service] + watchfiles==1.0.5 [e2e-llm-inference-service] + websocket-client==1.8.0 [e2e-llm-inference-service] + websockets==15.0.1 [e2e-llm-inference-service] + yarl==1.20.0 [e2e-llm-inference-service] Audited 1 package in 48ms [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Now using project "kserve" on server "https://a598157d2c94f4659a8dcc831855073a-75975bfcf6c3c9ed.elb.us-east-1.amazonaws.com:6443". [e2e-llm-inference-service] [e2e-llm-inference-service] You can add applications to this project with the 'new-app' command. For example, try: [e2e-llm-inference-service] [e2e-llm-inference-service] oc new-app rails-postgresql-example [e2e-llm-inference-service] [e2e-llm-inference-service] to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: [e2e-llm-inference-service] [e2e-llm-inference-service] kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname [e2e-llm-inference-service] [e2e-llm-inference-service] [INFO] Installing Kustomize v5.8.1 for linux/amd64... [e2e-llm-inference-service] [INFO] Kustomize v5.8.1 is already installed in /workspace/source/bin (>= v5.8.1) [e2e-llm-inference-service] make: Entering directory '/workspace/source' [e2e-llm-inference-service] make: Leaving directory '/workspace/source' [e2e-llm-inference-service] Creating namespace openshift-keda... [e2e-llm-inference-service] namespace/openshift-keda created [e2e-llm-inference-service] Namespace openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating OperatorGroup openshift-keda... [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-keda created [e2e-llm-inference-service] OperatorGroup openshift-keda created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Creating Subscription for openshift-custom-metrics-autoscaler-operator... [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-custom-metrics-autoscaler-operator created [e2e-llm-inference-service] Subscription openshift-custom-metrics-autoscaler-operator created/ensured. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Waiting for openshift-custom-metrics-autoscaler-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (15/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (20/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-custom-metrics-autoscaler-operator... (25/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (35/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (40/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (45/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 found, but not yet Succeeded (Phase: Installing). Waiting... (50/600) [e2e-llm-inference-service] CSV custom-metrics-autoscaler.v2.19.0-1 is ready (Phase: Succeeded). [e2e-llm-inference-service] --- [e2e-llm-inference-service] Applying KedaController custom resource... [e2e-llm-inference-service] Warning: resource kedacontrollers/keda is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] kedacontroller.keda.sh/keda configured [e2e-llm-inference-service] KedaController custom resource applied. [e2e-llm-inference-service] --- [e2e-llm-inference-service] Allowing time for KEDA components to be provisioned by the operator ... [e2e-llm-inference-service] Waiting for KEDA Operator pod (selector: "app=keda-operator") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-operator" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-operator" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-operator" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-operator-68bff59c-bjfxd 1/1 Running 0 41s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-operator" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-operator-68bff59c-bjfxd condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-operator" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Operator pod is ready. [e2e-llm-inference-service] Waiting for KEDA Metrics API Server pod (selector: "app=keda-metrics-apiserver") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-metrics-apiserver" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-metrics-apiserver" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-metrics-apiserver-559f75f947-fztjc 1/1 Running 0 46s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-metrics-apiserver-559f75f947-fztjc condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-metrics-apiserver" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Metrics API Server pod is ready. [e2e-llm-inference-service] Waiting for KEDA Webhook pod (selector: "app=keda-admission-webhooks") to be ready in namespace openshift-keda... [e2e-llm-inference-service] Waiting for pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" to be created... [e2e-llm-inference-service] Pod -l "app=keda-admission-webhooks" in namespace "openshift-keda" found. [e2e-llm-inference-service] Current pods for -l "app=keda-admission-webhooks" in namespace "openshift-keda": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] keda-admission-5fbd5c4644-ssckd 1/1 Running 0 51s [e2e-llm-inference-service] Waiting up to 120s for pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" to become ready... [e2e-llm-inference-service] pod/keda-admission-5fbd5c4644-ssckd condition met [e2e-llm-inference-service] Pod(s) -l "app=keda-admission-webhooks" in namespace "openshift-keda" are ready. [e2e-llm-inference-service] KEDA Webhook pod is ready. [e2e-llm-inference-service] --- [e2e-llm-inference-service] ✅ KEDA deployment script finished successfully. [e2e-llm-inference-service] KSERVE_CONTROLLER_IMAGE=quay.io/opendatahub/kserve-controller@sha256:4027d5c68e4b38dbdf70fa5c894b50b863d7857990c7284124c7a020c312a97c [e2e-llm-inference-service] LLMISVC_CONTROLLER_IMAGE=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:e069958436927d1ba06d07865aae89800ab14a8c639ffa5751e1c39f1132cc2d [e2e-llm-inference-service] KSERVE_AGENT_IMAGE=quay.io/opendatahub/kserve-agent@sha256:eb91db6613fb4bde95991e8e02d444548e2e6e23bd5e4e0a11c1f350f3203183 [e2e-llm-inference-service] KSERVE_ROUTER_IMAGE=quay.io/opendatahub/kserve-router@sha256:78561fce36825084e3c48d0137c513c345701cffde73e206abb14ec219f8181b [e2e-llm-inference-service] STORAGE_INITIALIZER_IMAGE=quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb [e2e-llm-inference-service] Installing KServe via kustomize... [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] # Warning: 'commonLabels' is deprecated. Please use 'labels' instead. Run 'kustomize edit fix' to update your Kustomization automatically. [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for CRDs to be established... [e2e-llm-inference-service] Waiting for CRD inferenceservices.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD inferenceservices.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD llminferenceserviceconfigs.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD llminferenceserviceconfigs.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD clusterstoragecontainers.serving.kserve.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD clusterstoragecontainers.serving.kserve.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io condition met [e2e-llm-inference-service] Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD datascienceclusters.datasciencecluster.opendatahub.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met [e2e-llm-inference-service] Applying all resources... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/clusterstoragecontainers.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/dscinitializations.dscinitialization.opendatahub.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencegraphs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepoolimports.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.x-k8s.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/inferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceserviceconfigs.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/llminferenceservices.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/servingruntimes.serving.kserve.io serverside-applied [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/trainedmodels.serving.kserve.io serverside-applied [e2e-llm-inference-service] serviceaccount/kserve-controller-manager serverside-applied [e2e-llm-inference-service] serviceaccount/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-leader-election-role serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] role.rbac.authorization.k8s.io/llmisvc-leader-election-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-admin serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-edit serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-inferenceservice-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-distro-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-llmisvc-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-manager-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-metrics-reader-cluster-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-proxy-role serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-view serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/kserve-llmisvcconfig-read-access serverside-applied [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/llmisvc-leader-election-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-inferenceservice-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-llmisvc-distro-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-manager-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/kserve-proxy-rolebinding serverside-applied [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/llmisvc-manager-rolebinding serverside-applied [e2e-llm-inference-service] configmap/inferenceservice-config serverside-applied [e2e-llm-inference-service] configmap/kserve-parameters serverside-applied [e2e-llm-inference-service] secret/kserve-webhook-server-secret serverside-applied [e2e-llm-inference-service] secret/mlpipeline-s3-artifact serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-metrics-service serverside-applied [e2e-llm-inference-service] service/kserve-controller-manager-service serverside-applied [e2e-llm-inference-service] service/kserve-webhook-server-service serverside-applied [e2e-llm-inference-service] service/llmisvc-controller-manager-service serverside-applied [e2e-llm-inference-service] service/llmisvc-webhook-server-service serverside-applied [e2e-llm-inference-service] service/s3-service serverside-applied [e2e-llm-inference-service] deployment.apps/kserve-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/llmisvc-controller-manager serverside-applied [e2e-llm-inference-service] deployment.apps/seaweedfs serverside-applied [e2e-llm-inference-service] networkpolicy.networking.k8s.io/kserve-controller-manager serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-inferenceservice-image-volume-scc serverside-applied [e2e-llm-inference-service] securitycontextconstraints.security.openshift.io/openshift-ai-llminferenceservice-scc serverside-applied [e2e-llm-inference-service] clusterstoragecontainer.serving.kserve.io/default serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-amd-rocm-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-ppc64le-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-s390x-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-ibm-spyre-x86-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-intel-gaudi-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda-fast-1 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template-nvidia-cuda-fast-2 serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferencegraph.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/inferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceservice.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/llminferenceserviceconfig.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/servingruntime.serving.kserve.io serverside-applied [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/trainedmodel.serving.kserve.io serverside-applied [e2e-llm-inference-service] Waiting for llmisvc-controller-manager to be ready... [e2e-llm-inference-service] Waiting for pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to be created... [e2e-llm-inference-service] Pod -l "control-plane=llmisvc-controller-manager" in namespace "kserve" found. [e2e-llm-inference-service] Current pods for -l "control-plane=llmisvc-controller-manager" in namespace "kserve": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] llmisvc-controller-manager-69fcb569d9-h8mg4 0/1 Running 0 6s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" to become ready... [e2e-llm-inference-service] pod/llmisvc-controller-manager-69fcb569d9-h8mg4 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=llmisvc-controller-manager" in namespace "kserve" are ready. [e2e-llm-inference-service] Re-applying LLMInferenceServiceConfig resources with webhook validation... [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-decode-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-decode-worker-data-parallel serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-template serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-prefill-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-prefill-worker-data-parallel serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-router-route serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-scheduler is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-scheduler serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-template is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-template serverside-applied [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-tracing serverside-applied [e2e-llm-inference-service] Warning: modifying well-known config kserve/kserve-config-llm-worker-data-parallel is not recommended. Consider creating a custom config instead [e2e-llm-inference-service] llminferenceserviceconfig.serving.kserve.io/kserve-config-llm-worker-data-parallel serverside-applied [e2e-llm-inference-service] Applying DSC/DSCI resources... [e2e-llm-inference-service] dscinitialization.dscinitialization.opendatahub.io/test-dsci created [e2e-llm-inference-service] datasciencecluster.datasciencecluster.opendatahub.io/test-dsc created [e2e-llm-inference-service] KServe manual installation complete [e2e-llm-inference-service] 🔧 Configuration: [e2e-llm-inference-service] KServe deployment: ❌ disabled [e2e-llm-inference-service] Kuadrant deployment: ✅ enabled [e2e-llm-inference-service] [e2e-llm-inference-service] Checking OpenShift server version...(4.21.22) [e2e-llm-inference-service] 🎯 Server version (4.21.22) is 4.19.9 or higher - continue with the script [e2e-llm-inference-service] ⏳ Installing cert-manager [e2e-llm-inference-service] namespace/cert-manager-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] subscription.operators.coreos.com/openshift-cert-manager-operator created [e2e-llm-inference-service] Waiting for openshift-cert-manager-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription openshift-cert-manager-operator... (10/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 found, but not yet Succeeded (Phase: Installing). Waiting... (15/300) [e2e-llm-inference-service] CSV cert-manager-operator.v1.19.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD certificates.cert-manager.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD certificates.cert-manager.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io condition met [e2e-llm-inference-service] ✅ Cert-manager installed [e2e-llm-inference-service] ⏳ Installing openshift-lws-operator [e2e-llm-inference-service] namespace/openshift-lws-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] subscription.operators.coreos.com/leader-worker-set created [e2e-llm-inference-service] Waiting for leader-worker-set CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (0/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (5/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (10/300) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription leader-worker-set... (15/300) [e2e-llm-inference-service] CSV leader-worker-set.v1.0.0 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD leaderworkersetoperators.operator.openshift.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD leaderworkersetoperators.operator.openshift.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/leaderworkersetoperators.operator.openshift.io condition met [e2e-llm-inference-service] leaderworkersetoperator.operator.openshift.io/cluster created [e2e-llm-inference-service] ⏳ waiting for openshift-lws-operator to be ready.… [e2e-llm-inference-service] Waiting for pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to be created... [e2e-llm-inference-service] Pod -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" found. [e2e-llm-inference-service] Current pods for -l "name=openshift-lws-operator" in namespace "openshift-lws-operator": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-lws-operator-fd8ccff4c-ktmkv 1/1 Running 0 13s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" to become ready... [e2e-llm-inference-service] pod/openshift-lws-operator-fd8ccff4c-ktmkv condition met [e2e-llm-inference-service] Pod(s) -l "name=openshift-lws-operator" in namespace "openshift-lws-operator" are ready. [e2e-llm-inference-service] ✅ openshift-lws-operator installed [e2e-llm-inference-service] gatewayclass.gateway.networking.k8s.io/openshift-default created [e2e-llm-inference-service] Waiting for pod -l "app=istiod" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "app=istiod" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "app=istiod" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] istiod-openshift-gateway-94bb8fbfd-kqnw7 1/1 Running 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "app=istiod" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/istiod-openshift-gateway-94bb8fbfd-kqnw7 condition met [e2e-llm-inference-service] Pod(s) -l "app=istiod" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Creating a Gateway [e2e-llm-inference-service] Error from server (AlreadyExists): namespaces "openshift-ingress" already exists [e2e-llm-inference-service] ⏳ Creating gateway memory ConfigMap for parametersRef (2Gi) [e2e-llm-inference-service] configmap/gateway-proxy-config created [e2e-llm-inference-service] gateway.gateway.networking.k8s.io/openshift-ai-inference created [e2e-llm-inference-service] Waiting for pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to be created... [e2e-llm-inference-service] Pod -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" found. [e2e-llm-inference-service] Current pods for -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] openshift-ai-inference-openshift-default-655f7c58b4-57g8x 0/1 ContainerCreating 0 5s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" to become ready... [e2e-llm-inference-service] pod/openshift-ai-inference-openshift-default-655f7c58b4-57g8x condition met [e2e-llm-inference-service] Pod(s) -l "serving.kserve.io/gateway=kserve-ingress-gateway" in namespace "openshift-ingress" are ready. [e2e-llm-inference-service] ⏳ Installing RHCL(Kuadrant) operator [e2e-llm-inference-service] namespace/kuadrant-system created [e2e-llm-inference-service] subscription.operators.coreos.com/rhcl-operator created [e2e-llm-inference-service] operatorgroup.operators.coreos.com/kuadrant created [e2e-llm-inference-service] Waiting for rhcl-operator CSV to become ready... [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (0/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (5/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (10/600) [e2e-llm-inference-service] Waiting for CSV to be installed for subscription rhcl-operator... (15/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (20/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (25/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 found, but not yet Succeeded (Phase: Installing). Waiting... (30/600) [e2e-llm-inference-service] CSV rhcl-operator.v1.4.1 is ready (Phase: Succeeded). [e2e-llm-inference-service] Waiting for CRD kuadrants.kuadrant.io to appear (timeout: 90s)… [e2e-llm-inference-service] CRD kuadrants.kuadrant.io detected — waiting for it to become Established (timeout: 90s)… [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/kuadrants.kuadrant.io condition met [e2e-llm-inference-service] Waiting for apiserver discovery /apis/kuadrant.io/v1beta1 to list kuadrants (timeout: 120s)… [e2e-llm-inference-service] Discovery for kuadrant.io/v1beta1 includes kuadrants. [e2e-llm-inference-service] ⏳ sleeping 30s after discovery (RESTMapper can trail discovery)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant created [e2e-llm-inference-service] ⏳ waiting for Kuadrant Ready (attempt 1/2, timeout 5m)… [e2e-llm-inference-service] kuadrant.kuadrant.io/kuadrant condition met [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-s4vc7 1/1 Running 0 76s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-s4vc7 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ⏳ waiting for authorino service to be created... [e2e-llm-inference-service] service/authorino-authorino-authorization condition met [e2e-llm-inference-service] service/authorino-authorino-authorization annotated [e2e-llm-inference-service] Warning: resource authorinos/authorino is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. [e2e-llm-inference-service] authorino.operator.authorino.kuadrant.io/authorino configured [e2e-llm-inference-service] Waiting for pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" to be created... [e2e-llm-inference-service] Pod -l "control-plane=authorino-operator" in namespace "kuadrant-system" found. [e2e-llm-inference-service] Current pods for -l "control-plane=authorino-operator" in namespace "kuadrant-system": [e2e-llm-inference-service] NAME READY STATUS RESTARTS AGE [e2e-llm-inference-service] authorino-operator-6d85f6564-s4vc7 1/1 Running 0 86s [e2e-llm-inference-service] Waiting up to 600s for pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" to become ready... [e2e-llm-inference-service] pod/authorino-operator-6d85f6564-s4vc7 condition met [e2e-llm-inference-service] Pod(s) -l "control-plane=authorino-operator" in namespace "kuadrant-system" are ready. [e2e-llm-inference-service] ✅ kuadrant(authorino) installed [e2e-llm-inference-service] Patching ingress domain... [e2e-llm-inference-service] configmap/inferenceservice-config patched [e2e-llm-inference-service] pod "kserve-controller-manager-67485586b-hs5ft" deleted [e2e-llm-inference-service] Waiting for kserve-controller-manager to be ready... [e2e-llm-inference-service] pod/kserve-controller-manager-67485586b-v6pj6 condition met [e2e-llm-inference-service] Installing ODH Model Controller manually... [e2e-llm-inference-service] customresourcedefinition.apiextensions.k8s.io/accounts.nim.opendatahub.io created [e2e-llm-inference-service] serviceaccount/model-serving-api created [e2e-llm-inference-service] serviceaccount/odh-model-controller created [e2e-llm-inference-service] role.rbac.authorization.k8s.io/leader-election-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-editor-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/account-viewer-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/kserve-prometheus-k8s created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/metrics-reader created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/odh-model-controller-role created [e2e-llm-inference-service] clusterrole.rbac.authorization.k8s.io/proxy-role created [e2e-llm-inference-service] rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/model-serving-api created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/odh-model-controller-rolebinding-opendatahub created [e2e-llm-inference-service] clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created [e2e-llm-inference-service] configmap/odh-model-controller-parameters created [e2e-llm-inference-service] service/model-serving-api created [e2e-llm-inference-service] service/odh-model-controller-metrics-service created [e2e-llm-inference-service] service/odh-model-controller-webhook-service created [e2e-llm-inference-service] deployment.apps/model-serving-api created [e2e-llm-inference-service] deployment.apps/odh-model-controller created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/model-serving-api-metrics created [e2e-llm-inference-service] servicemonitor.monitoring.coreos.com/odh-model-controller-metrics-monitor created [e2e-llm-inference-service] template.template.openshift.io/guardrails-detector-huggingface-serving-template created [e2e-llm-inference-service] template.template.openshift.io/kserve-ovms created [e2e-llm-inference-service] template.template.openshift.io/mlserver-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cpu-x86-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-cuda-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-gaudi-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-multinode-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-rocm-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-ppc64le-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-s390x-runtime-template-fast-2 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-1 created [e2e-llm-inference-service] template.template.openshift.io/vllm-spyre-x86-runtime-template-fast-2 created [e2e-llm-inference-service] mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] validatingwebhookconfiguration.admissionregistration.k8s.io/validating.odh-model-controller.opendatahub.io created [e2e-llm-inference-service] Waiting for deployment "odh-model-controller" rollout to finish: 0 of 1 updated replicas are available... [e2e-llm-inference-service] deployment "odh-model-controller" successfully rolled out [e2e-llm-inference-service] networkpolicy.networking.k8s.io/allow-all created [e2e-llm-inference-service] KServe setup complete (namespace: kserve) [e2e-llm-inference-service] Add testing models to SeaweedFS S3 storage ... [e2e-llm-inference-service] Waiting for SeaweedFS deployment to be ready... [e2e-llm-inference-service] deployment "seaweedfs" successfully rolled out [e2e-llm-inference-service] S3 init job not completed, re-creating... [e2e-llm-inference-service] job.batch/s3-init replaced [e2e-llm-inference-service] Waiting for S3 init job to complete... [e2e-llm-inference-service] job.batch/s3-init condition met [e2e-llm-inference-service] Prepare CI namespace and install ServingRuntimes [e2e-llm-inference-service] Setting up CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Tearing down CI namespace: kserve-ci-e2e-test [e2e-llm-inference-service] Namespace kserve-ci-e2e-test does not exist, skipping deletion [e2e-llm-inference-service] CI namespace teardown complete [e2e-llm-inference-service] Creating namespace kserve-ci-e2e-test [e2e-llm-inference-service] namespace/kserve-ci-e2e-test created [e2e-llm-inference-service] Applying S3 artifact secret [e2e-llm-inference-service] secret/mlpipeline-s3-artifact created [e2e-llm-inference-service] Applying storage-config secret [e2e-llm-inference-service] secret/storage-config created [e2e-llm-inference-service] Applying SeaweedFS S3 credentials secret [e2e-llm-inference-service] secret/seaweedfs-s3-creds created [e2e-llm-inference-service] Linking seaweedfs-s3-creds to default service account [e2e-llm-inference-service] Creating odh-trusted-ca-bundle configmap [e2e-llm-inference-service] configmap/odh-trusted-ca-bundle created [e2e-llm-inference-service] Installing ServingRuntimes [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-autogluonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-huggingfaceserver-multinode created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-lgbserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-mlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-paddleserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-pmmlserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-predictiveserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-sklearnserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tensorflow-serving created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-torchserve created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-tritonserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-vllmserver created [e2e-llm-inference-service] servingruntime.serving.kserve.io/kserve-xgbserver created [e2e-llm-inference-service] CI namespace setup complete [e2e-llm-inference-service] Setup complete [e2e-llm-inference-service] === E2E cluster / operator summary === [e2e-llm-inference-service] Client Version: 4.20.11 [e2e-llm-inference-service] Kustomize Version: v5.6.0 [e2e-llm-inference-service] Server Version: 4.21.22 [e2e-llm-inference-service] Kubernetes Version: v1.34.8 [e2e-llm-inference-service] ClusterVersion desired: 4.21.22 [e2e-llm-inference-service] ClusterVersion history (latest): 4.21.22 (Completed) [e2e-llm-inference-service] CSVs in kuadrant-system: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-keda: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] custom-metrics-autoscaler.v2.19.0-1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in cert-manager-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-lws-operator: [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] cert-manager-operator.v1.19.0 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] leader-worker-set.v1.0.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] CSVs in openshift-operators (ODH / shared operators, filtered): [e2e-llm-inference-service] authorino-operator.v1.4.1 Succeeded [e2e-llm-inference-service] dns-operator.v1.4.0 Succeeded [e2e-llm-inference-service] limitador-operator.v1.4.0 Succeeded [e2e-llm-inference-service] rhcl-operator.v1.4.1 Succeeded [e2e-llm-inference-service] Kuadrant / Authorino (diagnostics): [e2e-llm-inference-service] CRD kuadrants.kuadrant.io versions: v1beta1 served=true storage=true [e2e-llm-inference-service] Subscriptions in kuadrant-system: [e2e-llm-inference-service] authorino-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators authorino-operator.v1.4.1 [e2e-llm-inference-service] dns-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators dns-operator.v1.4.0 [e2e-llm-inference-service] limitador-operator-stable-redhat-operators-openshift-marketplace stable redhat-operators limitador-operator.v1.4.0 [e2e-llm-inference-service] rhcl-operator stable redhat-operators rhcl-operator.v1.4.1 [e2e-llm-inference-service] Kuadrant CR conditions (kuadrant/kuadrant-system): [e2e-llm-inference-service] Ready=True (Ready) [e2e-llm-inference-service] KServe deployments in kserve: [e2e-llm-inference-service] kserve-controller-manager: ready=1 image=quay.io/opendatahub/kserve-controller@sha256:4027d5c68e4b38dbdf70fa5c894b50b863d7857990c7284124c7a020c312a97c [e2e-llm-inference-service] imageID: quay.io/opendatahub/kserve-controller@sha256:4027d5c68e4b38dbdf70fa5c894b50b863d7857990c7284124c7a020c312a97c [e2e-llm-inference-service] odh-model-controller: ready=1 image=quay.io/opendatahub/odh-model-controller:fast [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-model-controller@sha256:7e360c28fce3f1aacde49fe9cf4c5d6acb49591c5f98be12b0c2e09f8f4a2b86 [e2e-llm-inference-service] llmisvc-controller-manager: ready=1 image=quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:e069958436927d1ba06d07865aae89800ab14a8c639ffa5751e1c39f1132cc2d [e2e-llm-inference-service] imageID: quay.io/opendatahub/odh-kserve-llmisvc-controller@sha256:b7e9edb14d30d27374bc12fced7e7a0bc5df461f247fb0b69dde5b104ff0bb11 [e2e-llm-inference-service] === End E2E cluster / operator summary === [e2e-llm-inference-service] /workspace/source [e2e-llm-inference-service] CA certificate extracted [e2e-llm-inference-service] REQUESTS_CA_BUNDLE=/tmp/ca.crt [e2e-llm-inference-service] Run E2E tests: llminferenceservice and cluster_cpu and not autoscaling and not tracing [e2e-llm-inference-service] Starting E2E functional tests ... [e2e-llm-inference-service] Parallelism requested for pytest is 2 [e2e-llm-inference-service] ============================= test session starts ============================== [e2e-llm-inference-service] platform linux -- Python 3.11.13, pytest-7.4.4, pluggy-1.5.0 -- /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] cachedir: .pytest_cache [e2e-llm-inference-service] metadata: {'Python': '3.11.13', 'Platform': 'Linux-5.14.0-427.115.1.el9_4.x86_64-x86_64-with-glibc2.34', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.5.0'}, 'Plugins': {'asyncio': '0.23.8', 'json-report': '1.5.0', 'xdist': '3.6.1', 'httpx': '0.30.0', 'anyio': '4.9.0', 'metadata': '3.1.1', 'cov': '5.0.0'}, 'PLATFORM': 'el9'} [e2e-llm-inference-service] rootdir: /workspace/source/test/e2e [e2e-llm-inference-service] configfile: pytest.ini [e2e-llm-inference-service] plugins: asyncio-0.23.8, json-report-1.5.0, xdist-3.6.1, httpx-0.30.0, anyio-4.9.0, metadata-3.1.1, cov-5.0.0 [e2e-llm-inference-service] asyncio: mode=Mode.STRICT [e2e-llm-inference-service] created: 2/2 workers [e2e-llm-inference-service] 2 workers [42 items] [e2e-llm-inference-service] [e2e-llm-inference-service] scheduling tests via WorkStealingScheduling [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] 2026-07-01 20:58:31.262 5769 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-01 20:58:31.263 5766 kserve INFO [conftest.py:configure_logger():40] Logger configured [e2e-llm-inference-service] 2026-07-01 20:58:31.311 5766 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 20:58:31.311 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 20:58:31.314 5766 kserve.trace Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 20:58:31.314 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 20:58:31.321 5766 kserve.trace ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 20:58:31.321 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-with-section-name] [e2e-llm-inference-service] llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] 2026-07-01 20:58:54.801 5766 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 20:58:54.801 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 20:58:54.816 5766 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 20:58:54.816 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_gateway_section_name.py::test_gateway_section_name_propagation[cluster_single_node-cluster_cpu-without-section-name] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_enabled_requires_token[cluster_cpu-cluster_single_node-auth-enabled-default] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_invalid_token_rejected[cluster_cpu-cluster_single_node-auth-invalid-token] [e2e-llm-inference-service] llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_auth.py::test_llm_auth_disabled_no_token_required[cluster_cpu-cluster_single_node-auth-disabled] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] 2026-07-01 21:08:21.843 5766 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:08:21.843 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:08:21.856 5766 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 21:08:21.856 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] 2026-07-01 21:14:06.390 5766 kserve.trace Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:14:06.390 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:14:06.405 5766 kserve.trace ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 21:14:06.405 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():57] ✓ Successfully updated Gateway router-gateway-1 [e2e-llm-inference-service] 2026-07-01 21:14:06.405 5766 kserve.trace Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:14:06.405 5766 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-1 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:14:06.409 5766 kserve.trace Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-01 21:14:06.409 5766 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-01 21:14:06.423 5766 kserve.trace ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-01 21:14:06.423 5766 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-1 [e2e-llm-inference-service] 2026-07-01 21:14:06.424 5766 kserve.trace Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:14:06.424 5766 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:14:06.429 5766 kserve.trace Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-01 21:14:06.429 5766 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-01 21:14:06.441 5766 kserve.trace ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] 2026-07-01 21:14:06.441 5766 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-2 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] 2026-07-01 21:26:06.545 5766 kserve.trace Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:26:06.545 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():34] Checking Gateway router-gateway-2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:26:06.550 5766 kserve.trace Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-01 21:26:06.550 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():62] Resource not found, creating Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-01 21:26:06.557 5766 kserve.trace ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-01 21:26:06.557 5766 kserve.trace INFO [gw_api.py:create_or_update_gateway():70] ✓ Successfully created Gateway router-gateway-2 [e2e-llm-inference-service] 2026-07-01 21:26:06.557 5766 kserve.trace Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:26:06.557 5766 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-3 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:26:06.563 5766 kserve.trace Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-01 21:26:06.563 5766 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-01 21:26:06.575 5766 kserve.trace ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-01 21:26:06.575 5766 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-3 [e2e-llm-inference-service] 2026-07-01 21:26:06.576 5766 kserve.trace Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:26:06.576 5766 kserve.trace INFO [gw_api.py:create_or_update_route():121] Checking HttpRoute router-route-4 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] 2026-07-01 21:26:06.580 5766 kserve.trace Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-01 21:26:06.580 5766 kserve.trace INFO [gw_api.py:create_or_update_route():149] Resource not found, creating HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-01 21:26:06.589 5766 kserve.trace ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] 2026-07-01 21:26:06.589 5766 kserve.trace INFO [gw_api.py:create_or_update_route():157] ✓ Successfully created HttpRoute router-route-4 [e2e-llm-inference-service] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha1_to_v1alpha2_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_v1alpha2_to_v1alpha1_conversion [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_criticality_preservation_via_annotations [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_lora_criticality_preservation [e2e-llm-inference-service] llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_conversion.py::TestLLMInferenceServiceConversion::test_round_trip_conversion_preserves_fields [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-single-lora-adapter-hf] [e2e-llm-inference-service] llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_lora_adapters.py::test_llm_with_lora_adapters[cluster_cpu-multiple-lora-adapters] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_prestop_hook.py::test_prestop_hook[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_rolling_upgrade.py::test_rolling_upgrade_coordination[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_storage_version_migration.py::TestStorageVersionMigration::test_storage_version_migration_after_simulated_upgrade [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw0] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] [gw1] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [gw1] PASSED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] [e2e-llm-inference-service] =================================== FAILURES =================================== [e2e-llm-inference-service] _ test_llm_inference_service[router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] _ [e2e-llm-inference-service] [gw1] linux -- Python 3.11.13 /workspace/source/python/kserve/.venv/bin/python [e2e-llm-inference-service] [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] @pytest.mark.asyncio(loop_scope="session") [e2e-llm-inference-service] @pytest.mark.parametrize( [e2e-llm-inference-service] "test_case", [e2e-llm-inference-service] [ [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-gateway-ref", [e2e-llm-inference-service] "router-with-managed-route", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="custom-route-timeout-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="router-with-refs-test", [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[0], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[0]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[0], ROUTER_ROUTES[1]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=["router-managed", "workload-pd-cpu", "model-fb-opt-125m"], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-custom-route-timeout-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="custom-route-timeout-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-with-refs-pd", [e2e-llm-inference-service] "scheduler-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="You are an expert in Kubernetes-native machine learning serving platforms, with deep knowledge of the KServe project. " [e2e-llm-inference-service] "Explain the challenges of serving large-scale models, GPU scheduling, and how KServe integrates with capabilities like multi-model serving. " [e2e-llm-inference-service] "Provide a detailed comparison with open source alternatives, focusing on operational trade-offs.", [e2e-llm-inference-service] service_name="router-with-refs-pd-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] expected_gateway=ROUTER_GATEWAYS[1], [e2e-llm-inference-service] before_test=[ [e2e-llm-inference-service] lambda: create_router_resources( [e2e-llm-inference-service] gateways=[ROUTER_GATEWAYS[1]], [e2e-llm-inference-service] routes=[ROUTER_ROUTES[2], ROUTER_ROUTES[3]], [e2e-llm-inference-service] ) [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.custom_gateway, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-dp-ep-gpu", [e2e-llm-inference-service] "workload-dp-ep-prefill-gpu", [e2e-llm-inference-service] "model-deepseek-v2-lite", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="Delve into the multifaceted implications of a fully disaggregated cloud architecture, specifically " [e2e-llm-inference-service] "where the compute plane (P) and the data plane (D) are independently deployed and managed for a " [e2e-llm-inference-service] "geographically distributed, high-throughput, low-latency microservices ecosystem. Beyond the " [e2e-llm-inference-service] "fundamental challenges of network latency and data consistency, elaborate on the advanced " [e2e-llm-inference-service] "considerations and trade-offs inherent in such a setup: 1. Network Architecture and Protocols: " [e2e-llm-inference-service] "How would the network fabric and underlying protocols (e.g., RDMA, custom transport layers) need to " [e2e-llm-inference-service] "evolve to support optimal performance and minimize inter-plane communication overhead, especially for " [e2e-llm-inference-service] "synchronous operations? Discuss the role of network programmability (e.g., SDN, P4) in dynamically " [e2e-llm-inference-service] "optimizing routing and traffic flow between P and D. 2. Advanced Data Consistency and Durability: " [e2e-llm-inference-service] "Explore sophisticated data consistency models (e.g., causal consistency, strong eventual consistency) " [e2e-llm-inference-service] "and their applicability in balancing performance and data integrity across a globally distributed data plane. " [e2e-llm-inference-service] "Detail strategies for ensuring data durability and fault tolerance, including multi-region replication, " [e2e-llm-inference-service] "intelligent partitioning, and recovery mechanisms in the event of partial or full plane failures. " [e2e-llm-inference-service] "3. Dynamic Resource Orchestration and Cost Optimization: Analyze how an orchestration layer would intelligently " [e2e-llm-inference-service] "manage the independent scaling of compute (P) and data (D) resources, considering fluctuating workloads, " [e2e-llm-inference-service] "cost efficiency, and performance targets (e.g., using predictive analytics for resource provisioning). " [e2e-llm-inference-service] "Discuss mechanisms for dynamically reallocating compute nodes to different data partitions based on " [e2e-llm-inference-service] "workload patterns and data locality, potentially involving live migration strategies. " [e2e-llm-inference-service] "4. Security and Compliance in a Distributed Landscape: Address the enhanced security perimeter " [e2e-llm-inference-service] "challenges, including securing communication channels between P and D (encryption in transit, mutual TLS), " [e2e-llm-inference-service] "fine-grained access control to data at rest and in motion, and identity management across disaggregated " [e2e-llm-inference-service] "components. Discuss how such an architecture impacts compliance with regulatory frameworks (e.g., GDPR, HIPAA) " [e2e-llm-inference-service] "concerning data sovereignty, privacy, and auditability. 5. Operational Complexity and Observability: " [e2e-llm-inference-service] "Examine the increased complexity in monitoring, logging, and tracing across highly decoupled compute and " [e2e-llm-inference-service] "data planes. What specialized tooling and practices (e.g., distributed tracing with OpenTelemetry, advanced AIOps) " [e2e-llm-inference-service] "would be essential? How would incident response and troubleshooting differ in this disaggregated environment " [e2e-llm-inference-service] "compared to traditional integrated systems? Consider the challenges of pinpointing root causes across " [e2e-llm-inference-service] "independent failures. 6. Real-world Applicability and Future Trends: Identify specific industries " [e2e-llm-inference-service] "or use cases (e.g., high-frequency trading, IoT edge processing, large language model inference) " [e2e-llm-inference-service] "where the benefits of P/D disaggregation would strongly outweigh its complexities. " [e2e-llm-inference-service] "Conclude by speculating on emerging technologies or paradigms (e.g., serverless compute functions " [e2e-llm-inference-service] "directly interacting with object storage, in-memory disaggregation) that could further drive or " [e2e-llm-inference-service] "transform P/D disaggregation in cloud computing.", [e2e-llm-inference-service] max_tokens=2000, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_gpu, [e2e-llm-inference-service] pytest.mark.cluster_nvidia, [e2e-llm-inference-service] pytest.mark.cluster_nvidia_roce, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-no-scheduler", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.no_scheduler, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-fb-opt-125m", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="This test simulates DP+EP that can run on CPU, the idea is to test the LWS-based deployment, " [e2e-llm-inference-service] "but without the resources requirements for DP+EP (GPUs and ROCe/IB).", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_multi_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler config tests [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-inline-config-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Chat completions endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] model_name="Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="choices"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-configmap-ref", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-configmap-ref-test", [e2e-llm-inference-service] before_test=[create_scheduler_configmap], [e2e-llm-inference-service] after_test=[delete_scheduler_configmap], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-replicas", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-ha-replicas-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-custom-template", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-custom-template-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[pytest.mark.cluster_cpu, pytest.mark.cluster_single_node], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Scheduler v0.6 → v0.7 migration tests. [e2e-llm-inference-service] # Deploy v0.6-style configs and verify the controller migrates them [e2e-llm-inference-service] # so the v0.7 scheduler boots successfully. [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-pd-config-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-pd-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-v06-nonzero-threshold-migration", [e2e-llm-inference-service] "workload-llmd-simulator-pd", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="scheduler-v06-threshold-migration-test", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Precise prefix KV cache routing test [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "scheduler-with-precise-prefix-cache-inline-config", [e2e-llm-inference-service] "workload-llmd-simulator-kvcache", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] service_name="precise-prefix-cache-test", [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Models endpoint coverage [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=create_response_assertion(with_field="data"), [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/chat/completions [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches("facebook/opt-125m"), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] peers=[ [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-llmd-simulator", [e2e-llm-inference-service] "model-qwen2.5-0.5b", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/chat/completions", [e2e-llm-inference-service] prompt="What is KServe?", [e2e-llm-inference-service] payload_formatter=chat_completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] "Qwen/Qwen2.5-0.5B-Instruct" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/Qwen/Qwen2.5-0.5B-Instruct", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.llmd_simulator, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — LoRA adapter [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/completions", [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] model_name=f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] payload_formatter=completions_payload, [e2e-llm-inference-service] response_assertion=assert_model_field_matches( [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1" [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # Model-based routing via X-Gateway-Model-Name header — /v1/models (base + LoRA) [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-fb-opt-125m-with-lora-hf", [e2e-llm-inference-service] ], [e2e-llm-inference-service] endpoint="/v1/models", [e2e-llm-inference-service] response_assertion=assert_models_contains( [e2e-llm-inference-service] "facebook/opt-125m", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] "lora-adapter-1", [e2e-llm-inference-service] f"publishers/{KSERVE_TEST_NAMESPACE}/models/lora-adapter-1", [e2e-llm-inference-service] ), [e2e-llm-inference-service] url_getter=get_model_routing_url, [e2e-llm-inference-service] extra_headers={ [e2e-llm-inference-service] MODEL_ROUTING_HEADER: f"publishers/{KSERVE_TEST_NAMESPACE}/models/facebook/opt-125m", [e2e-llm-inference-service] }, [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.model_routing, [e2e-llm-inference-service] pytest.mark.lora, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] # PVC storage tests -- validate direct PVC volume mount with real vLLM serving [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-single-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-pd-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] response_assertion=assert_200_with_choices, [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_single_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] pytest.param( [e2e-llm-inference-service] TestCase( [e2e-llm-inference-service] base_refs=[ [e2e-llm-inference-service] "router-managed", [e2e-llm-inference-service] "workload-simulated-dp-ep-cpu", [e2e-llm-inference-service] "model-pvc", [e2e-llm-inference-service] ], [e2e-llm-inference-service] prompt="KServe is a", [e2e-llm-inference-service] before_test=[ensure_pvc_with_model], [e2e-llm-inference-service] ), [e2e-llm-inference-service] marks=[ [e2e-llm-inference-service] pytest.mark.cluster_cpu, [e2e-llm-inference-service] pytest.mark.cluster_multi_node, [e2e-llm-inference-service] pytest.mark.pvc_storage, [e2e-llm-inference-service] ], [e2e-llm-inference-service] ), [e2e-llm-inference-service] ], [e2e-llm-inference-service] indirect=["test_case"], [e2e-llm-inference-service] ids=generate_test_id, [e2e-llm-inference-service] ) [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def test_llm_inference_service(test_case: TestCase): # noqa: F811 [e2e-llm-inference-service] inject_k8s_proxy() [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = KServeClient( [e2e-llm-inference-service] config_file=os.environ.get("KUBECONFIG", "~/.kube/config"), [e2e-llm-inference-service] client_configuration=client.Configuration(), [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] service_name = test_case.llm_service.metadata.name [e2e-llm-inference-service] if not test_case.llm_service.metadata.annotations: [e2e-llm-inference-service] test_case.llm_service.metadata.annotations = {} [e2e-llm-inference-service] [e2e-llm-inference-service] test_case.llm_service.metadata.annotations[ [e2e-llm-inference-service] "security.opendatahub.io/enable-auth" [e2e-llm-inference-service] ] = "false" [e2e-llm-inference-service] prefix = test_case.log_prefix [e2e-llm-inference-service] [e2e-llm-inference-service] test_failed = False [e2e-llm-inference-service] try: [e2e-llm-inference-service] print(f"{prefix} Creating LLMInferenceService {service_name}") [e2e-llm-inference-service] create_llmisvc(kserve_client, test_case.llm_service) [e2e-llm-inference-service] print(f"{prefix} Waiting for LLMInferenceService {service_name} to be ready") [e2e-llm-inference-service] wait_for_llm_isvc_ready( [e2e-llm-inference-service] kserve_client, test_case.llm_service, test_case.wait_timeout [e2e-llm-inference-service] ) [e2e-llm-inference-service] print(f"{prefix} Waiting for model response from {service_name}") [e2e-llm-inference-service] > wait_for_model_response( [e2e-llm-inference-service] kserve_client, [e2e-llm-inference-service] test_case, [e2e-llm-inference-service] test_case.wait_timeout, [e2e-llm-inference-service] extra_headers=test_case.extra_headers, [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:816: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] args = (, TestCase(base_refs=['router-managed', 'scheduler-v0... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900) [e2e-llm-inference-service] kwargs = {'extra_headers': None}, func_name = 'wait_for_model_response' [e2e-llm-inference-service] timestamp_start = '2026-07-01T22:04:08.612532', start_time = 1782943448.6129894 [e2e-llm-inference-service] duration = 1274.1102931499481, timestamp_end = '2026-07-01T22:25:22.723285' [e2e-llm-inference-service] [e2e-llm-inference-service] @functools.wraps(func) [e2e-llm-inference-service] def wrapper(*args, **kwargs): [e2e-llm-inference-service] func_name = func.__name__ [e2e-llm-inference-service] [e2e-llm-inference-service] timestamp_start = datetime.now().isoformat() [e2e-llm-inference-service] logger.info( [e2e-llm-inference-service] f"[{func_name}] [{timestamp_start}] start - args={args}, kwargs={kwargs}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] start_time = time.time() [e2e-llm-inference-service] [e2e-llm-inference-service] try: [e2e-llm-inference-service] > result = func(*args, **kwargs) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/logging.py:40: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] kserve_client = [e2e-llm-inference-service] test_case = TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KSer... {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m') [e2e-llm-inference-service] timeout_seconds = 900, extra_headers = None [e2e-llm-inference-service] [e2e-llm-inference-service] @log_execution [e2e-llm-inference-service] def wait_for_model_response( [e2e-llm-inference-service] kserve_client: KServeClient, [e2e-llm-inference-service] test_case: TestCase, # noqa: F811 [e2e-llm-inference-service] timeout_seconds: int = 900, [e2e-llm-inference-service] extra_headers: Optional[Dict[str, str]] = None, [e2e-llm-inference-service] ) -> str: [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] [e2e-llm-inference-service] > response = wait_for(get_successful_response, timeout=timeout_seconds, interval=5.0) [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1119: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] assertion_fn = .get_successful_response at 0x7eff80ebb1a0> [e2e-llm-inference-service] timeout = 900, interval = 5.0 [e2e-llm-inference-service] [e2e-llm-inference-service] def wait_for( [e2e-llm-inference-service] assertion_fn: Callable[[], Any], timeout: float = 5.0, interval: float = 0.1 [e2e-llm-inference-service] ) -> Any: [e2e-llm-inference-service] """Wait for the assertion to succeed within timeout.""" [e2e-llm-inference-service] deadline = time.time() + timeout [e2e-llm-inference-service] last_msg = None [e2e-llm-inference-service] while True: [e2e-llm-inference-service] try: [e2e-llm-inference-service] > return assertion_fn() [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1215: [e2e-llm-inference-service] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [e2e-llm-inference-service] [e2e-llm-inference-service] def get_successful_response(): [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_case.url_getter: [e2e-llm-inference-service] service_url = test_case.url_getter(kserve_client, test_case.llm_service) [e2e-llm-inference-service] else: [e2e-llm-inference-service] service_url = get_llm_service_url(kserve_client, test_case.llm_service) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to get service URL: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] model_url = service_url + test_case.endpoint [e2e-llm-inference-service] [e2e-llm-inference-service] headers = {"Content-Type": "application/json"} [e2e-llm-inference-service] if extra_headers: [e2e-llm-inference-service] headers.update(extra_headers) [e2e-llm-inference-service] [e2e-llm-inference-service] if test_case.payload_formatter is not None: [e2e-llm-inference-service] test_payload = test_case.payload_formatter(test_case) [e2e-llm-inference-service] elif test_case.prompt is not None: [e2e-llm-inference-service] test_payload = { [e2e-llm-inference-service] "model": test_case.model_name [e2e-llm-inference-service] if not extra_headers or MODEL_ROUTING_HEADER not in extra_headers [e2e-llm-inference-service] else extra_headers[MODEL_ROUTING_HEADER], [e2e-llm-inference-service] "prompt": test_case.prompt, [e2e-llm-inference-service] "max_tokens": test_case.max_tokens, [e2e-llm-inference-service] } [e2e-llm-inference-service] else: [e2e-llm-inference-service] test_payload = None [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Calling LLM service at {model_url} with payload {test_payload}") [e2e-llm-inference-service] try: [e2e-llm-inference-service] if test_payload is not None: [e2e-llm-inference-service] response = post_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] json_data=test_payload, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] else: [e2e-llm-inference-service] response = get_with_retry( [e2e-llm-inference-service] model_url, [e2e-llm-inference-service] headers=headers, [e2e-llm-inference-service] timeout=test_case.response_timeout, [e2e-llm-inference-service] ) [e2e-llm-inference-service] except Exception as e: [e2e-llm-inference-service] logger.error(f"❌ Failed to call model: {e}") [e2e-llm-inference-service] raise AssertionError(f"❌ Failed to call model: {e}") from e [e2e-llm-inference-service] [e2e-llm-inference-service] logger.info(f"Model response is {response.status_code}: {response.text[:500]}") [e2e-llm-inference-service] [e2e-llm-inference-service] if 200 <= response.status_code < 300: [e2e-llm-inference-service] return response [e2e-llm-inference-service] > raise AssertionError( [e2e-llm-inference-service] f"Service returned {response.status_code}: {response.text}" [e2e-llm-inference-service] ) [e2e-llm-inference-service] E AssertionError: Service returned 502: [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:1115: AssertionError [e2e-llm-inference-service] ------------------------------ Captured log setup ------------------------------ [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1653 Checking LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b in namespace kserve-ci-e2e-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1679 Resource not found, creating LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1689 ✓ Successfully created LLMInferenceServiceConfig workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] ------------------------------ Captured log call ------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [test_llm_inference_service] [2026-07-01T22:03:22.441446] start - args=(), kwargs={'test_case': TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': None, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m')} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:fixtures.py:1704 No HTTP proxy configured for k8s client [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [create_llmisvc] [2026-07-01T22:03:22.454097] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [create_llmisvc] [2026-07-01T22:03:22.533862] end - ✅ in 0.080s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_llm_isvc_ready] [2026-07-01T22:03:22.534008] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, 900), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: No conditions found in status [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-01T22:03:29Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'severity': 'Info', 'status': 'False', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'message': 'Inference Pool kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool exists but no Gateway controller has accepted it yet', 'reason': 'WaitingForGateway', 'severity': 'Info', 'status': 'False', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'message': 'The following HTTPRoutes are not ready: [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: "False" (reason "InvalidKind", message "referencing unsupported backendRef: group \\"inference.networking.x-k8s.io\\" kind \\"InferencePool\\"")]', 'reason': 'HTTPRoutesNotReady', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'message': 'Deployment rollout in progress', 'reason': 'Progressing', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'reason': 'Progressing', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-01T22:03:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready', 'WorkloadsReady'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-01T22:03:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: Missing true conditions: {'RouterReady', 'Ready'}, expected {'RouterReady', 'Ready', 'WorkloadsReady'}, got [{'lastTransitionTime': '2026-07-01T22:03:35Z', 'severity': 'Info', 'status': 'True', 'type': 'HTTPRoutesReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'severity': 'Info', 'status': 'True', 'type': 'InferencePoolReady'}, {'lastTransitionTime': '2026-07-01T22:03:38Z', 'severity': 'Info', 'status': 'True', 'type': 'MainWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:37Z', 'severity': 'Info', 'status': 'True', 'type': 'PrefillWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:29Z', 'severity': 'Info', 'status': 'True', 'type': 'PresetsCombined'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'Ready'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'status': 'False', 'type': 'RouterReady'}, {'lastTransitionTime': '2026-07-01T22:03:35Z', 'message': 'Deployment does not have minimum availability.', 'reason': 'MinimumReplicasUnavailable', 'severity': 'Info', 'status': 'False', 'type': 'SchedulerWorkloadReady'}, {'lastTransitionTime': '2026-07-01T22:03:38Z', 'status': 'True', 'type': 'WorkloadsReady'}] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [wait_for_llm_isvc_ready] [2026-07-01T22:04:08.612379] end - ✅ in 46.078s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [wait_for_model_response] [2026-07-01T22:04:08.612532] start - args=(, TestCase(base_refs=['router-managed', 'scheduler-v06-pd-config-migration', 'workload-llmd-simulator-pd'], prompt='KServe is a', service_name='scheduler-v06-pd-migration-test', endpoint='/v1/completions', max_tokens=20, payload_formatter=None, response_assertion=, wait_timeout=900, response_timeout=60, extra_headers=None, url_getter=None, expected_gateway=None, before_test=[], after_test=[], peers=[], llm_service={'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}, model_name='facebook/opt-125m'), 900), kwargs={'extra_headers': None} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-01T22:04:08.612993] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-01T22:04:08.621089] end - ✅ in 0.008s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1108 ❌ Failed to call model: HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions (Caused by ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1222 Waiting: ❌ Failed to call model: HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Max retries exceeded with url: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions (Caused by ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:34 [get_llm_service_url] [2026-07-01T22:12:18.091352] start - args=(, {'api_version': 'serving.kserve.io/v1alpha1', [e2e-llm-inference-service] 'kind': 'LLMInferenceService', [e2e-llm-inference-service] 'metadata': {'annotations': {'security.opendatahub.io/enable-auth': 'false'}, [e2e-llm-inference-service] 'creation_timestamp': None, [e2e-llm-inference-service] 'deletion_grace_period_seconds': None, [e2e-llm-inference-service] 'deletion_timestamp': None, [e2e-llm-inference-service] 'finalizers': None, [e2e-llm-inference-service] 'generate_name': None, [e2e-llm-inference-service] 'generation': None, [e2e-llm-inference-service] 'labels': None, [e2e-llm-inference-service] 'managed_fields': None, [e2e-llm-inference-service] 'name': 'scheduler-v06-pd-migration-test', [e2e-llm-inference-service] 'namespace': 'kserve-ci-e2e-test', [e2e-llm-inference-service] 'owner_references': None, [e2e-llm-inference-service] 'resource_version': None, [e2e-llm-inference-service] 'self_link': None, [e2e-llm-inference-service] 'uid': None}, [e2e-llm-inference-service] 'spec': {'baseRefs': [{'name': 'router-managed-scheduler-v06-pd-5a5debb2'}, [e2e-llm-inference-service] {'name': 'scheduler-v06-pd-config-migrati-bb3264a2'}, [e2e-llm-inference-service] {'name': 'workload-llmd-simulator-pd-sche-64d0fc8b'}]}, [e2e-llm-inference-service] 'status': None}), kwargs={} [e2e-llm-inference-service] INFO e2e.llmisvc.logging:logging.py:43 [get_llm_service_url] [2026-07-01T22:12:18.154059] end - ✅ in 0.062s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1092 Calling LLM service at http://aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions with payload {'model': 'facebook/opt-125m', 'prompt': 'KServe is a', 'max_tokens': 20} [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] WARNING urllib3.connectionpool:connectionpool.py:868 Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com', port=80): Read timed out. (read timeout=60)")': /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1111 Model response is 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:1219 Timed out waiting: Service returned 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [wait_for_model_response] [2026-07-01T22:25:22.723285] end - ❌ 1274.110s: Service returned 502: [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:test_llm_inference_service.py:831 [router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] ❌ ERROR: Failed to call llm inference service scheduler-v06-pd-migration-test: Service returned 502: [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1240 🔍 # Diagnostics for 'scheduler-v06-pd-migration-test' in 'kserve-ci-e2e-test' [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1241 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1242 # LLMInferenceService scheduler-v06-pd-migration-test [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1245 apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] security.opendatahub.io/enable-auth: 'false' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:22Z' [e2e-llm-inference-service] finalizers: [e2e-llm-inference-service] - serving.kserve.io/llmisvc-finalizer [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:security.opendatahub.io/enable-auth: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:baseRefs: {} [e2e-llm-inference-service] manager: OpenAPI-Generator [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:22Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:finalizers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] v:"serving.kserve.io/llmisvc-finalizer": {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:22Z' [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:addresses: {} [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-decode-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-prefill-worker-data-parallel: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-router-route: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-scheduler: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-template: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-tracing: {} [e2e-llm-inference-service] f:serving.kserve.io/config-llm-worker-data-parallel: {} [e2e-llm-inference-service] f:appliedConfigs: {} [e2e-llm-inference-service] f:conditions: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:router: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:gateways: {} [e2e-llm-inference-service] f:scheduler: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:inferencePool: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:service: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:url: {} [e2e-llm-inference-service] f:workloads: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:prefill: {} [e2e-llm-inference-service] f:primary: {} [e2e-llm-inference-service] f:scheduler: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:04:07Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] resourceVersion: '82330' [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] baseRefs: [e2e-llm-inference-service] - name: router-managed-scheduler-v06-pd-5a5debb2 [e2e-llm-inference-service] - name: scheduler-v06-pd-config-migrati-bb3264a2 [e2e-llm-inference-service] - name: workload-llmd-simulator-pd-sche-64d0fc8b [e2e-llm-inference-service] model: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uri: '' [e2e-llm-inference-service] status: [e2e-llm-inference-service] addresses: [e2e-llm-inference-service] - name: gateway-external-model-routing [e2e-llm-inference-service] url: http://aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com/ [e2e-llm-inference-service] - name: gateway-external [e2e-llm-inference-service] url: http://aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] - name: gateway-internal-model-routing [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/ [e2e-llm-inference-service] - name: gateway-internal [e2e-llm-inference-service] url: http://openshift-ai-inference-openshift-default.openshift-ingress.svc.cluster.local/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-template: kserve-config-llm-decode-template [e2e-llm-inference-service] serving.kserve.io/config-llm-decode-worker-data-parallel: kserve-config-llm-decode-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-template: kserve-config-llm-prefill-template [e2e-llm-inference-service] serving.kserve.io/config-llm-prefill-worker-data-parallel: kserve-config-llm-prefill-worker-data-parallel [e2e-llm-inference-service] serving.kserve.io/config-llm-router-route: kserve-config-llm-router-route [e2e-llm-inference-service] serving.kserve.io/config-llm-scheduler: kserve-config-llm-scheduler [e2e-llm-inference-service] serving.kserve.io/config-llm-template: kserve-config-llm-template [e2e-llm-inference-service] serving.kserve.io/config-llm-tracing: kserve-config-llm-tracing [e2e-llm-inference-service] serving.kserve.io/config-llm-worker-data-parallel: kserve-config-llm-worker-data-parallel [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: HTTPRoutesReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: InferencePoolReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: MainWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:37Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PrefillWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:29Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: PresetsCombined [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:04:07Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Ready [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:04:07Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: RouterReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:04:07Z' [e2e-llm-inference-service] severity: Info [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: SchedulerWorkloadReady [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: WorkloadsReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] url: http://aa082f3b5080b4f778116aa87581940a-1302445891.us-east-1.elb.amazonaws.com/kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:44 TIME NAMESPACE SOURCE TYPE REASON MESSAGE [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:45 -------------------------------------------------------------------------------------------------- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-68bf46444d-nz2fk to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:56 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:56 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:56 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.37:8000/health": dial tcp 10.132.0.37:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:18 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.37:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-68bf46444d-nz2fk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:54 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:54 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:54 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-disabled-test-kserve-router-scheduler-9bc4884b5-shml5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-router-scheduler-9bc4884b5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-disabled-test-kserve-68bf46444d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy auth-disabled-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "auth-disabled-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-disabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-disabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-disabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-disabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-disabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-disabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-disabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:56 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-disabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-disabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-d8474687d-xkkfx to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:54 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulling Pulling image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Successfully pulled image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" in 38.539s (38.539s including waiting). Image size: 3531177328 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:13 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.32:8000/health": dial tcp 10.132.0.32:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-d8474687d-xkkfx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-64d4b65cc6-hlfjg to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:44 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:44 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:44 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.013s (1.013s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning Unhealthy Readiness probe failed: timeout: failed to connect service "10.134.0.48:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-enabled-test-kserve-router-scheduler-64d4b65cc6-hlfjg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-router-scheduler-64d4b65cc6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-enabled-test-kserve-d8474687d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-enabled-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-enabled-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-enabled-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-enabled-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-enabled-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-enabled-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-enabled-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-enabled-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-enabled-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-6f6986f4-vmpkj to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:25 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:25 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:25 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:10 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:37 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.35:8000/health": dial tcp 10.132.0.35:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-6f6986f4-vmpkj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-54958d97466ldts to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-router-scheduler-54958d9746 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set auth-invalid-token-test-kserve-6f6986f4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/auth-invalid-token-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/auth-invalid-token-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/auth-invalid-token-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/auth-invalid-token-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/auth-invalid-token-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/auth-invalid-token-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/auth-invalid-token-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [auth-invalid-token-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-auth-invalid-token-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:37 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-criticality": Operation cannot be fulfilled on llminferenceservices.serving.kserve.io "conv-test-criticality": the object has been modified; please apply your changes to the latest version and try again [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:37 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-lora-crit": llminferenceservices.serving.kserve.io "conv-test-lora-crit" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:38 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-round-trip": llminferenceservices.serving.kserve.io "conv-test-round-trip" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:37 kserve-ci-e2e-test LLMInferenceServiceController Warning UpdateFailed Failed to update status for LLMInferenceService "conv-test-v1a1-to-v1a2": llminferenceservices.serving.kserve.io "conv-test-v1a1-to-v1a2" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:24:26 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.50:8001/health": dial tcp 10.132.0.50:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-bf6648c8f-l76d2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:11 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:11 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:11 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:24:26 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.51:8000/health": dial tcp 10.132.0.51:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-prefill-84bf95b496-tchc8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-prefill-84bf95b496 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-pd-test-kserve-router-scheduler-65466ct4gx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-65466ct4gx to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:45 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:45 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-router-scheduler-6546688786 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-pd-test-kserve-bf6648c8f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-pd-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-pd-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:24:37 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.42/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.42:8000/health": dial tcp 10.132.0.42:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-bbd5bdc5b-t6rkv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: custom-route-timeout-test-kserve-router-scheduler-5599744dttprw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5599744dttprw to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:42 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-router-scheduler-5599744d7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set custom-route-timeout-test-kserve-bbd5bdc5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:38 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy custom-route-timeout-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "custom-route-timeout-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/custom-route-timeout-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/custom-route-timeout-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/custom-route-timeout-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/custom-route-timeout-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/custom-route-timeout-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/custom-route-timeout-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/custom-route-timeout-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:42 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [custom-route-timeout-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-custom-route-timeout-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/e2e-pvc-model-download-9dx26 to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:03 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:57 kserve-ci-e2e-test job-controller Normal SuccessfulCreate Created pod: e2e-pvc-model-download-9dx26 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:17 kserve-ci-e2e-test job-controller Normal Completed Job completed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:57 kserve-ci-e2e-test persistentvolume-controller Normal WaitForFirstConsumer waiting for first consumer to be created before binding [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:58 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-564866f9b6-tqnr4_f420500d-b6c1-4335-9b84-0352757a07ad Normal Provisioning External provisioner is provisioning volume for claim "kserve-ci-e2e-test/e2e-pvc-model-storage" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:58 kserve-ci-e2e-test persistentvolume-controller Normal ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:00 kserve-ci-e2e-test ebs.csi.aws.com_aws-ebs-csi-driver-controller-564866f9b6-tqnr4_f420500d-b6c1-4335-9b84-0352757a07ad Normal ProvisioningSucceeded Successfully provisioned volume pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc52ftwt [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc52ftwt to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:53 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:54 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.423s (1.423s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:54 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:54 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:09 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:25 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.46:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bpz57q [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5bpz57q to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:31 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67nm9lm to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.57/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:37 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.57:8000/health": dial tcp 10.132.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:32:54 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:10 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.57:8000/health": dial tcp 10.132.0.57:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c86ddcf67 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-3c960099-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-3c960099-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:47 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:56 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-3c960099] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.46/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 779ms (779ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.46:8001/health": dial tcp 10.132.0.46:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:39 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:39 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8-mvk8h [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.47/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:11 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:11 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:11 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:19:56 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.47:8000/health": dial tcp 10.132.0.47:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:39 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f7cvrx [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5f85dddfb9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-50bc673d-kserve-bb84757f8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:05 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:17 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-50bc673d] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:38 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test00d7278d8a22c4e39146a6b0eb840f45-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846qq7vl to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.40/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:13 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:13 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:13 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:57 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.40:8000/health": dial tcp 10.132.0.40:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.40:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-87882a8e-kserve-55fbb5c846 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:08 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-87882a8e] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test21fe6730fe484f3a92b1a16afe1bac8f-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.59/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:35:37 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.59:8000/health": dial tcp 10.132.0.59:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:37 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.59:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 85976476fb for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 in StatefulSet llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:35:47 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:25 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:15 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-dc21cb14] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test386bd5808c5c450f11fd8632e1f821fb-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d48hqx6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d48hqx6 to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.29/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:27 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:27 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:27 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:39 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:24 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:38 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-route-e95b1dc1] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:58 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:58 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:58 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:44 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.39:8000/health": dial tcp 10.132.0.39:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7-cdtzl [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewzkwf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler-597fc788f8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewzkwf to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:54 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:54 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:54 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:44 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:44 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-7ca60146-kserve-ffbdb7db7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:46 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:08:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-7ca60146] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test5216bfd716f919dc046bc693ceb22e41-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.41/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.41:8000/health": dial tcp 10.132.0.41:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:37 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b-q2jnj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2vbzm to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:14 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:14 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:14 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:37 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:37 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2vbzm [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler-65cf48d687 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7bcb69f67b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:18 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:16 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:10:17 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:33 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-fb-opt-125m-with-ba4d693a] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:13:37 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test05addb65ba05195619f26ef266e8fc04-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:00 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.53/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:02 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:12 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedToRetrieveImagePullSecret Unable to retrieve some image pull secrets (llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-ph46t); attempting to pull the image may not succeed. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:28 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedMount MountVolume.SetUp failed for volume "tls-certs" : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:18 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:00 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:02 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:02 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:02 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.52:8000/health": dial tcp 10.132.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:16 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.52:8000/health": dial tcp 10.132.0.52:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:18 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.52:8000/health": context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test leaderworkerset Normal CreatingRevision Creating revision with key 686d468674 for a newly created LeaderWorkerSet [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created leader statefulset llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Replicas are progressing, with 0 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test statefulset-controller Normal SuccessfulCreate create Pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 in StatefulSet llmisvc-model-pvc-router-manage-2577e794-kserve-mn successful [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test leaderworkerset Normal GroupsProgressing Created worker statefulset for leader pod llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:12 kserve-ci-e2e-test leaderworkerset Normal AllGroupsReady All replicas are ready, with 1 groups ready of total 1 groups [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:53 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-2577e794-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-2577e794-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-scc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.LeaderWorkerSet kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:57 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:25:47 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-2577e794] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:01 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test37591b20e96e9663d45a730d03070f1e-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:54 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:55 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.45:8000/health": dial tcp 10.132.0.45:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.45:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:49 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-59b9d263-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-59b9d263-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:15:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:15:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:15:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:10 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-59b9d263] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:24 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testc9569cf4801efc0ed27b2f25ffaee875-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:01 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.48/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.48:8001/health": dial tcp 10.132.0.48:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:01 kserve-ci-e2e-test attachdetach-controller Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:07 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.49:8000/health": dial tcp 10.132.0.49:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79497db4cc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:55 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-pvc-router-manage-e8706282-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-pvc-router-manage-e8706282-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:20 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-pvc-router-manage-e8706282] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:27 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testf5d060a5eb39a04e074b78907a1556a6-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bzrdv [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bzrdv to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:29 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:41 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:22 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:49:01 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4c1ba212] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:41 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test80f99418f0f3eb4c80f02e6144a6d4f2-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98ftfz2b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98ftfz2b to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.52/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:55 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:15 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:54 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-4f8c0978] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:55 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test76d7f532acb694e4a7bcef75d32cd8a1-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd85rfs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd85rfs to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.49/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:31 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.134.0.49:8000/health": EOF [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:36 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:23 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-model-qwen2-5-0-5b-rout-a50492e9] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testb7025ca4d8a6f8f5b2fd08b5581d2678-llmisvc-mode [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-kcm8z to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.27/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:15 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-kcm8z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:30 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-4b931143-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-4b931143-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:05:04 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-4b931143] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test8ac8e3d2264ccb939eb021b0b835847c-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xwdp7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xwdp7 to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.26/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:49 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:05 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.26:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-5b1e8f15-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-5b1e8f15-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:21 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-5b1e8f15] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test7f54e84970003a6e7372bdbcb574f7ed-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879h867m [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879h867m to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.25/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:01 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 29.647s (29.647s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:01 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:01 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:25 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:40 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.25:8000/health": dial tcp 10.133.0.25:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy llmisvc-router-managed-test-llm-e45d1f79-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route: HTTPRoute.gateway.networking.k8s.io "llmisvc-router-managed-test-llm-e45d1f79-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:53 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:16 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [llmisvc-router-managed-test-llm-e45d1f79] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-testef4d2875be14b30dc1561ed84d0d4bde-llmisvc-rout [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schednpfr to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.28/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:20 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:21 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.523s (1.523s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:22 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:22 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:55 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:55 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schednpfr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:06:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheduler-6fcb489785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheklks8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheduler-7dbcb75dbc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-scheklks8 to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.32/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:58 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:22:00 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0, llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:01 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:01 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmbsw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheduler-8445d86596 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmbsw to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:29 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:29 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.64/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:41 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche4f97z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:28 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scheduler-79866dc455 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schentcfc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheduler-78f7f9db6d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schentcfc to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:07 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:39 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:20:39 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:25 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:25 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:25 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche45kkn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheduler-5dd88bfbb7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejh6wz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scheduler-5f555d4d85 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schejh6wz to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:52 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:53 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:53 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:54 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d9kvx5 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:17:24 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8nfrs to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:08 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:08 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:09 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:09 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:09 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:12 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:11:12 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche8nfrs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:09:07 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheduler-7b78b68f76 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherk7cs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:40 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheduler-6d86bd4d9d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherk7cs to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:41 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.50/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:41 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:41 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:02:41 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:31 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:31 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:00 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheln89s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:00 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheduler-67b4bb9646 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheln89s to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:00 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:00 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:00 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:18:01 kserve-ci-e2e-test attachdetach-controller Warning FailedAttachVolume Multi-Attach error for volume "pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb" Volume is already used by pod(s) llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-6j9zb, llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8spp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:27 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:21:27 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4pfx2 to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:33 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.51/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:33 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:33 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:33 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:15 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:07:15 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4pfx2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:04:32 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheduler-68cc9685d6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:48 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheduler-749449dbc8 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewvvrj to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:49 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.34/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:49 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:49 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:01:49 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:49 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:03:49 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:57 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:57 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:57 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:04 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.62:8000/health": dial tcp 10.132.0.62:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:54:39 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-multiple-adapters-test-kserve-76d7f87f5d-zt9x2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-multiple-adapters-test-kserve-76d7f87f5d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:40 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-multiple-adapters-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-multiple-adapters-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-multiple-adapters-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:12 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:31 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-multiple-adapters-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/lora-single-adapter-test-kserve-6b64c6f8b9-2lb72 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:00 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:04 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:04 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:04 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:35:19 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.61:8000/health": dial tcp 10.132.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:14 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:46:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.61:8000/health": dial tcp 10.132.0.61:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: lora-single-adapter-test-kserve-6b64c6f8b9-2lb72 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set lora-single-adapter-test-kserve-6b64c6f8b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy lora-single-adapter-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "lora-single-adapter-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/lora-single-adapter-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.HTTPRoute kserve-ci-e2e-test/lora-single-adapter-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:59 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.DestinationRule kserve-ci-e2e-test/lora-single-adapter-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:34:02 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/lora-single-adapter-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:35:30 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [lora-single-adapter-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-5ms96 to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.45/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulling Pulling image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Successfully pulled image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" in 3.563s (3.563s including waiting). Image size: 300567477 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:01 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:01 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:01 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:00 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7f56466b78-t94t8 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.30/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:58 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:59 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" in 1.723s (1.723s including waiting). Image size: 98346788 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:59 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:59 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.30:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-7f56466b78-t94t8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-7f56466b78-5ms96 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf9pc8c7 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.31/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:44 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:44 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:44 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" in 1.45s (1.45s including waiting). Image size: 75073927 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:16 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-uds-tokenizer:vllm-v0.19.1" in 29.907s (29.907s including waiting). Image size: 2989890188 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:16 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:16 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:17 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:34 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Liveness probe failed: timeout: failed to connect service "10.132.0.31:9003" within 1s: context deadline exceeded [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container tokenizer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:00 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-router-scheduler-c6c4fdf96 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set precise-prefix-cache-test-kserve-7f56466b78 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy precise-prefix-cache-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "precise-prefix-cache-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:42 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/precise-prefix-cache-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/precise-prefix-cache-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/precise-prefix-cache-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:43 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/precise-prefix-cache-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/precise-prefix-cache-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/precise-prefix-cache-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/precise-prefix-cache-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:58 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [precise-prefix-cache-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:00:00 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-precise-prefix-cache-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-b8f4z to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.67:8000/health": dial tcp 10.132.0.67:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:06 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.67:8000/health": dial tcp 10.132.0.67:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-6769cdd744-kpvrw to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.68/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:55 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:21 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.68:8000/health": dial tcp 10.132.0.68:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-6769cdd744-b8f4z [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-6769cdd744-kpvrw [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-85465b59fc-5v5db to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:46 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:48 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:48 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:48 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:23 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:23 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: prestop-hook-test-kserve-router-scheduler-85465b59fc-5v5db [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set prestop-hook-test-kserve-router-scheduler-85465b59fc from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set prestop-hook-test-kserve-6769cdd744 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/prestop-hook-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/prestop-hook-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/prestop-hook-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/prestop-hook-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/prestop-hook-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/prestop-hook-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/prestop-hook-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/prestop-hook-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:59:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [prestop-hook-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:05 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [prestop-hook-test] is no longer Ready because of: MainWorkloadReady, WorkloadsReady [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-prestop-hook-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-r2wjc to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.69/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-54bc76f55b-r2wjc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:36 kserve-ci-e2e-test replicaset-controller Normal SuccessfulDelete Deleted pod: rolling-upgrade-test-kserve-54bc76f55b-r2wjc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-glml8 to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.36/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:52 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-859df57bcb-glml8 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-6cmlr to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.67/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:52 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:52 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:52 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-6cmlr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-54bc76f55b from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set rolling-upgrade-test-kserve-859df57bcb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:36 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled down replica set rolling-upgrade-test-kserve-54bc76f55b from 1 to 0 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:48 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy rolling-upgrade-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "rolling-upgrade-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/rolling-upgrade-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/rolling-upgrade-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/rolling-upgrade-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:00:51 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/rolling-upgrade-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/rolling-upgrade-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/rolling-upgrade-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:24 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [rolling-upgrade-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Deployment kserve-ci-e2e-test/rolling-upgrade-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-rolling-upgrade-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-ss4x6 to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:31 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulling Pulling image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:35 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Successfully pulled image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" in 3.122s (3.122s including waiting). Image size: 181251556 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:03 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:03 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:03 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:59:09 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.44:15021/healthz/ready": dial tcp 10.134.0.44:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-1-openshift-default-54c47cdfc7-ss4x6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-1-openshift-default-54c47cdfc7 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:31 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 20:58:35 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:06 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-j47sj to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:07 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.60/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:18 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:7d15cebf9b62f3f235c0eab5158ac8ff2fda86a1d193490dc94c301402c99da8" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:18 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:18 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container istio-proxy [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:23 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning Unhealthy Startup probe failed: Get "http://10.134.0.60:15021/healthz/ready": dial tcp 10.134.0.60:15021: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:06 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-gateway-2-openshift-default-684dc9b4b9-j47sj [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:06 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-gateway-2-openshift-default-684dc9b4b9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:06 kserve-ci-e2e-test service-controller Normal EnsuringLoadBalancer Ensuring load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:10 kserve-ci-e2e-test service-controller Normal EnsuredLoadBalancer Ensured load balancer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:06 kserve-ci-e2e-test gateway_labeler_controller Normal AddedLabel Added label istio.io/rev=openshift-gateway to gateway router-gateway-2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-88fcd5785-4gknp to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.54/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.54:8001/health": dial tcp 10.132.0.54:8001: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-88fcd5785-4gknp [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.55/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:35 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:35 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:35 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:01 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.55:8000/health": dial tcp 10.132.0.55:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-prefill-6f65776896-vzvsg [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-prefill-6f65776896 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-pd-test-kserve-router-scheduler-7b9956b84w4fkr [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-7b9956b84w4fkr to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.61/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:32 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:33 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:33 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:33 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:20 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-router-scheduler-7b9956b846 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-pd-test-kserve-88fcd5785 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-pd-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-pd-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-pd-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-pd-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:33 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-pd-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:26:44 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-pd-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:12 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-pd-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:29:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-pd-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-7948b44cf6-7jnwf to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.43/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:26 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.43:8000/health": dial tcp 10.132.0.43:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:40 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-7948b44cf6-7jnwf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.44/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:40 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:40 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: router-with-refs-test-kserve-router-scheduler-5594f94979-fqs8s [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-router-scheduler-5594f94979 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set router-with-refs-test-kserve-7948b44cf6 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/router-with-refs-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/router-with-refs-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:21 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/router-with-refs-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:22 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/router-with-refs-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:14:23 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/router-with-refs-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:15:09 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/router-with-refs-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:32 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [router-with-refs-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:16:40 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-router-with-refs-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-w4dpn to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.35/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:08 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:25 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.35:8000/health": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-744bf59fbd-w4dpn [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87brkb6 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.66/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:08 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-router-scheduler-68f876cc5d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-configmap-ref-test-kserve-744bf59fbd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-configmap-ref-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-configmap-ref-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:04 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:05 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-configmap-ref-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:20 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:56:37 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-configmap-ref-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:08 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-configmap-ref-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.73/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:36 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-5bc7bd6864-gd7l9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-custom-template-test-kserve-router-scheduler-567jsr2c [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567jsr2c to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.39/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:36 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:36 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-router-scheduler-567469f665 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-custom-template-test-kserve-5bc7bd6864 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:59 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-custom-template-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-custom-template-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-custom-template-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-custom-template-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-custom-template-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:03 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-custom-template-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-custom-template-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-custom-template-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:14 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-custom-template-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:35 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-custom-template-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:36 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-custom-template-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.70/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:49 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-5c5cfcf776-r296b [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhg8tc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhg8tc to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.37/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dmcsm9 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.71/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: service unhealthy (responded with "NOT_SERVING") [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dbc4 from 0 to 2 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:49 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-ha-replicas-test-kserve-5c5cfcf776 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:37 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-ha-replicas-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-ha-replicas-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:49 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:01:50 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-ha-replicas-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:02:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:02:06 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:02:07 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:02:22 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-ha-replicas-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:29 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-ha-replicas-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-c4czf to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.33/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:52 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:48:07 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.133.0.33:8000/health": dial tcp 10.133.0.33:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-7c6c58cc96-c4czf [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfgcgqs to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:52 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-router-scheduler-d9ddf6c4d from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-inline-config-test-kserve-7c6c58cc96 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:16 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-inline-config-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-inline-config-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-inline-config-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-inline-config-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-inline-config-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-inline-config-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-inline-config-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-inline-config-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:51 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-inline-config-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:47:52 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-inline-config-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.69/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulling Pulling image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Successfully pulled image "ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2" in 739ms (739ms including waiting). Image size: 44914394 bytes. [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container llm-d-routing-sidecar [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:28 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk to ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.133.0.38/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-inference-sim:v0.8.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test kubelet/ip-10-0-140-104.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.72/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set scheduler-v06-pd-migration-test-kserve-5c87756974 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:25 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy scheduler-v06-pd-migration-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "scheduler-v06-pd-migration-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:26 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-scheduler-v06-pd-migration-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:34 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:03:35 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 22:04:07 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [scheduler-v06-pd-migration-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-czrmh to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.56/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:50 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:35 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.56:8000/health": dial tcp 10.132.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:02 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.56:8000/health": dial tcp 10.132.0.56:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-8486474f84-dz7zq to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.58/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:27 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:13 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.58:8000/health": dial tcp 10.132.0.58:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:35 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:51 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.58:8000/health": dial tcp 10.132.0.58:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-8486474f84-czrmh [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-8486474f84-dz7zq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7bb5b6598f-9j9zq to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.63/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:23 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:24 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:24 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:24 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:35 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:33:35 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7bb5b6598f-cxj44 to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.62/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:47 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:13 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:13 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-7bb5b6598f-cxj44 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:22 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: stop-feature-test-kserve-router-scheduler-7bb5b6598f-9j9zq [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-7bb5b6598f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-router-scheduler-7bb5b6598f from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-8486474f84 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:22 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set stop-feature-test-kserve-8486474f84 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:20 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy stop-feature-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "stop-feature-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-stop-feature-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/stop-feature-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/stop-feature-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:45 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/stop-feature-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:28:54 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/stop-feature-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [stop-feature-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Secret kserve-ci-e2e-test/stop-feature-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ServiceAccount kserve-ci-e2e-test/stop-feature-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:46 kserve-ci-e2e-test LLMInferenceServiceController Warning Error Reconciliation failed: failed to reconcile networking: failed to reconcile scheduler: failed to build expected scheduler deployment: failed to attach model artifacts to scheduler deployment: secrets "stop-feature-test-epp-sa-dockercfg-5q9g5" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:30:47 kserve-ci-e2e-test LLMInferenceServiceController Warning LLMInferenceServiceNotReady LLMInferenceService [stop-feature-test] is no longer Ready because of: MainWorkloadReady, PrefillWorkerWorkloadReady, PrefillWorkloadReady, RouterReady, SchedulerWorkloadReady, WorkerWorkloadReady, WorkloadsReady [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Deployment kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.Service kserve-ci-e2e-test/stop-feature-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:31:13 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.InferencePool kserve-ci-e2e-test/stop-feature-test-inference-pool [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-654fdf69fb-mc965 to ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.132.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Pulled Container image "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.19.0" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:20 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:05 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Startup probe failed: Get "https://10.132.0.65:8000/health": dial tcp 10.132.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:19 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:35 kserve-ci-e2e-test kubelet/ip-10-0-138-219.ec2.internal Warning Unhealthy Readiness probe failed: Get "https://10.132.0.65:8000/health": dial tcp 10.132.0.65:8000: connect: connection refused [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-654fdf69fb-mc965 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 None kserve-ci-e2e-test Normal Scheduled Successfully assigned kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-58bcc64bb9-zrd4x to ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:16 kserve-ci-e2e-test multus Normal AddedInterface Add eth0 [10.134.0.65/23] from ovn-kubernetes [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:16 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "quay.io/opendatahub/kserve-storage-initializer@sha256:cc7b8fac71b36ebfd1661675b86a84ade93db501973e87758605bd27f47937cb" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:16 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:16 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container storage-initializer [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:17 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Pulled Container image "ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2" already present on machine [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:17 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Created Created container: main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:17 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Started Started container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:19 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Normal Killing Stopping container main [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:19 kserve-ci-e2e-test kubelet/ip-10-0-132-117.ec2.internal Warning FailedPreStopHook PreStopHook failed [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test replicaset-controller Normal SuccessfulCreate Created pod: tls-verification-test-kserve-router-scheduler-58bcc64bb9-zrd4x [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-router-scheduler-58bcc64bb9 from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test deployment-controller Normal ScalingReplicaSet Scaled up replica set tls-verification-test-kserve-654fdf69fb from 0 to 1 [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:10 kserve-ci-e2e-test OpenDataHubModelController Warning ReconcileError Failed to reconcile LLMInferenceService: 1 error occurred: * failed to get HTTPRoute for AuthPolicy tls-verification-test-kserve-route-authn: failed to get HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route: HTTPRoute.gateway.networking.k8s.io "tls-verification-test-kserve-route" not found [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/tls-verification-test-kserve-workload-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ServiceAccount kserve-ci-e2e-test/tls-verification-test-epp-sa [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Role kserve-ci-e2e-test/tls-verification-test-epp-role [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.RoleBinding kserve-ci-e2e-test/tls-verification-test-epp-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Deployment kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:15 kserve-ci-e2e-test LLMInferenceServiceController Normal Created Created v1.Service kserve-ci-e2e-test/tls-verification-test-epp-service [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Created (combined from similar events): Created v1.DestinationRule kserve-ci-e2e-test/tls-verification-test-kserve-shadow-svc [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:30 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.Secret kserve-ci-e2e-test/tls-verification-test-kserve-self-signed-certs [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:55:31 kserve-ci-e2e-test LLMInferenceServiceController Normal Updated Updated v1.HTTPRoute kserve-ci-e2e-test/tls-verification-test-kserve-route [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:16 kserve-ci-e2e-test LLMInferenceServiceController Normal LLMInferenceServiceReady LLMInferenceService [tls-verification-test] is Ready [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:56 2026-07-01 21:57:19 kserve-ci-e2e-test LLMInferenceServiceController Normal Deleted Deleted v1.ClusterRoleBinding /kserve-ci-e2e-test-tls-verification-test-epp-auth-rb [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### init-container 'llm-d-routing-sidecar' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 Flag --decoder-use-tls has been deprecated, use --enable-tls=decoder instead [e2e-llm-inference-service] Flag --prefiller-use-tls has been deprecated, use --enable-tls=prefiller instead [e2e-llm-inference-service] {"level":"info","ts":1782943407.5337932,"msg":"Proxy starting","Built on":"v0.9.0-rc.2","From Git SHA":"181aa8358916e19b8844ccc752b2d6153d4b2ad6"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.5338128,"msg":"Proxy configuration","config":"{\"Port\":\"8000\",\"KVConnector\":\"nixlv2\",\"ECConnector\":\"\",\"DataParallelSize\":1,\"MaxIdleConnsPerHost\":1024,\"EnablePrefillerSampling\":false,\"PrefillMaxRetries\":0,\"PrefillRetryBackoff\":200000000,\"UseTLSForPrefiller\":true,\"UseTLSForDecoder\":true,\"UseTLSForEncoder\":false,\"InsecureSkipVerifyForPrefiller\":false,\"InsecureSkipVerifyForEncoder\":false,\"InsecureSkipVerifyForDecoder\":false,\"SecureServing\":true,\"CertPath\":\"/var/run/kserve/tls\",\"MooncakeBootstrapPort\":8998,\"EnableSSRFProtection\":true,\"InferencePoolNamespace\":\"kserve-ci-e2e-test\",\"InferencePoolName\":\"scheduler-v06-pd-migration-test-inference-pool\",\"PoolGroup\":\"inference.networking.x-k8s.io\",\"DecodeChunkSize\":0,\"Tracing\":false,\"MoRIIOWriteMode\":false,\"MoRIIODecodeNotifyPort\":61005,\"MoRIIODecodeHandshakePort\":6301,\"MoRIIODecodePodIP\":\"\",\"MoRIIOParallelDispatch\":false,\"MoRIIOPrefillHandshakePort\":6301,\"MoRIIOPrefillNotifyPort\":61005,\"MoRIIOTPSize\":1,\"MoRIIODPSize\":1,\"MoRIIORemoteHosts\":null,\"MoRIIODPSizeLocal\":0,\"MoRIIODecodeHosts\":null,\"DecoderURL\":\"https://localhost:8001\"}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.5351279,"logger":"allowlist-validator","msg":"starting SSRF protection allowlist validator","namespace":"kserve-ci-e2e-test","poolName":"scheduler-v06-pd-migration-test-inference-pool","gvr":"inference.networking.x-k8s.io/v1alpha2, Resource=inferencepools"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.5491235,"logger":"allowlist-validator","msg":"InferencePool added","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.556143,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.38":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943407.556191,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":2,"targets":{"10.133.0.38":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943407.6360724,"logger":"allowlist-validator","msg":"allowlist validator started successfully"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.6375115,"logger":"proxy server on port 8000","msg":"server TLS configured"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.6375284,"logger":"proxy server on port 8000","msg":"starting","addr":"[::]:8000"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.9659019,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943408.9709668,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943416.587136,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943416.624406,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943416.8670585,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943417.0238123,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943418.0334244,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943437.5495262,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943437.5587413,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943437.5587907,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943467.549767,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943467.5598364,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943467.5598803,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943497.5502748,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943497.5594847,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943497.5595384,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:05:08 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943508.784905,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"ec747412-7598-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/01 22:05:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943512.8055215,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"eeda2d27-7598-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] 2026/07/01 22:05:20 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943520.8250873,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"f3a1cf60-7598-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1782943527.5509036,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943527.561364,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943527.5614245,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:05:36 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943536.847552,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"fd2e7f40-7598-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1782943557.5515258,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943557.5812597,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943557.5813134,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:06:08 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943568.8715086,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"1044f3b4-7599-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1782943587.551861,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943587.5613415,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943587.5614028,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943617.5527213,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943617.561832,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943617.5618806,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:07:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943632.9216924,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"367252d2-7599-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1782943647.5535262,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943647.5632336,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943647.5633109,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943677.5541058,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943677.5630364,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943677.5630877,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943707.5545607,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943707.563883,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943707.5639315,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943737.555621,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943737.5642946,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943737.5643523,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:09:12 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782943752.9786353,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"7e0188e0-7599-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1782943767.5563006,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943767.5665555,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943767.566609,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943797.5566282,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943797.565705,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943797.5657578,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943827.5570443,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943827.5671747,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943827.5672264,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943857.5578115,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943857.5688834,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943857.5689418,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943887.5587287,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943887.570387,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943887.5704477,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943917.5592194,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943917.5718954,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943917.571948,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943947.5593913,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943947.5677416,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943947.5677989,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943977.559763,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782943977.569026,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943977.569078,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944007.5605981,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944007.571192,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944007.5712733,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944037.5611036,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944037.6503162,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944037.650372,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944067.5614634,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944067.5708177,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944067.570877,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944097.5620532,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944097.5712183,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944097.5712729,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944127.5629017,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944127.572156,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944127.5722127,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944157.5632758,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944157.573023,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944157.573084,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944187.563359,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944187.5723853,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944187.572446,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944217.5639799,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944217.5737479,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944217.5738015,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944247.564984,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944247.5725875,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944247.5726418,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944277.5658524,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944277.5734658,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944277.573514,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944307.5664864,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944307.5776505,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944307.577703,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944337.567129,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944337.5765505,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944337.5766108,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944367.5679011,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944367.5772483,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944367.5773013,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944397.5684707,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944397.5783966,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944397.578462,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944427.5695002,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944427.578666,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944427.5787256,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944457.569971,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944457.581838,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944457.5818949,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944487.5709136,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944487.5802557,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944487.5803125,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944517.5716057,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944517.5826204,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944517.5826735,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944547.5723581,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944547.5818932,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944547.5819476,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944577.5725398,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944577.5833733,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944577.5834298,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:23:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782944602.6614563,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"7874deb2-759b-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] {"level":"info","ts":1782944607.5728657,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944607.5821128,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944607.582177,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944637.573803,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944637.580838,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944637.5808902,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944667.5748258,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944667.5851357,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944667.5851896,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944697.5749395,"logger":"allowlist-validator","msg":"InferencePool updated","name":"scheduler-v06-pd-migration-test-inference-pool"} [e2e-llm-inference-service] {"level":"info","ts":1782944697.584598,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782944697.5846567,"logger":"allowlist-validator","msg":"rebuilt allowlist","targetCount":4,"targets":{"10.133.0.38":{},"10.134.0.69":{},"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz":{},"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk":{}}} [e2e-llm-inference-service] 2026/07/01 22:25:22 http: proxy error: tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, not 10.133.0.38 [e2e-llm-inference-service] {"level":"error","ts":1782944722.7202296,"logger":"proxy server on port 8000","msg":"prefill request failed","request_id":"c0045f5a-759b-11f1-8614-0a580a860045","body":"","error":"prefill returned 502","stacktrace":"github.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).handleNIXLV2\n\t/workspace/pkg/sidecar/proxy/connector_nixlv2.go:242\ngithub.com/llm-d/llm-d-router/pkg/sidecar/proxy.(*Server).createRoutes.(*Server).disaggregatedPrefillHandler.func3\n\t/workspace/pkg/sidecar/proxy/chat_completions.go:183\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2861\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:178\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.68.0/handler.go:66\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2322\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:3340\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:2109"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0701 22:03:28.025760 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8001, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1782943408025341200, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0701 22:03:28.062379 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0701 22:03:28.067346 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0701 22:03:28.067407 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0701 22:03:28.067455 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0701 22:03:28.067697 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0701 22:03:28.067786 1 grpc.go:126] "Server starting" protocol="gRPC" port=8001 [e2e-llm-inference-service] I0701 22:03:28.068388 1 http.go:96] "Server starting" protocol="HTTPS" port=8001 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 I0701 22:03:26.796655 1 config.go:602] "Configuration:" =< [e2e-llm-inference-service] { [e2e-llm-inference-service] "IP": "", [e2e-llm-inference-service] "PodName": "", [e2e-llm-inference-service] "PodNameSpace": "", [e2e-llm-inference-service] "VllmDevMode": false, [e2e-llm-inference-service] "block-size": 16, [e2e-llm-inference-service] "data-parallel-rank": -1, [e2e-llm-inference-service] "data-parallel-size": 1, [e2e-llm-inference-service] "dataset-in-memory": false, [e2e-llm-inference-service] "dataset-path": "", [e2e-llm-inference-service] "dataset-table-name": "llmd", [e2e-llm-inference-service] "dataset-url": "", [e2e-llm-inference-service] "default-embedding-dimensions": 384, [e2e-llm-inference-service] "ec-transfer-config": "", [e2e-llm-inference-service] "enable-kvcache": false, [e2e-llm-inference-service] "enable-prefix-caching": false, [e2e-llm-inference-service] "enable-request-id-headers": false, [e2e-llm-inference-service] "enable-sleep-mode": false, [e2e-llm-inference-service] "enforce-eager": false, [e2e-llm-inference-service] "event-batch-size": 16, [e2e-llm-inference-service] "failure-injection-rate": 0, [e2e-llm-inference-service] "failure-types": null, [e2e-llm-inference-service] "fake-metrics": null, [e2e-llm-inference-service] "fake-metrics-refresh-interval": 100000000, [e2e-llm-inference-service] "global-cache-hit-threshold": 0, [e2e-llm-inference-service] "hash-seed": "", [e2e-llm-inference-service] "inter-token-latency": 0, [e2e-llm-inference-service] "inter-token-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-size": 1024, [e2e-llm-inference-service] "kv-cache-transfer-latency": 0, [e2e-llm-inference-service] "kv-cache-transfer-latency-std-dev": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-per-token": 0, [e2e-llm-inference-service] "kv-cache-transfer-time-std-dev": 0, [e2e-llm-inference-service] "latency-calculator": "", [e2e-llm-inference-service] "lora-modules": null, [e2e-llm-inference-service] "max-cpu-loras": 1, [e2e-llm-inference-service] "max-loras": 1, [e2e-llm-inference-service] "max-model-len": 1024, [e2e-llm-inference-service] "max-num-seqs": 5, [e2e-llm-inference-service] "max-tool-call-array-param-length": 5, [e2e-llm-inference-service] "max-tool-call-integer-param": 100, [e2e-llm-inference-service] "max-tool-call-number-param": 100, [e2e-llm-inference-service] "max-waiting-queue-length": 1000, [e2e-llm-inference-service] "min-tool-call-array-param-length": 1, [e2e-llm-inference-service] "min-tool-call-integer-param": 0, [e2e-llm-inference-service] "min-tool-call-number-param": 0, [e2e-llm-inference-service] "mm-encoder-only": false, [e2e-llm-inference-service] "mm-processor-kwargs": "", [e2e-llm-inference-service] "mode": "random", [e2e-llm-inference-service] "model": "facebook/opt-125m", [e2e-llm-inference-service] "object-tool-call-not-required-field-probability": 50, [e2e-llm-inference-service] "port": 8000, [e2e-llm-inference-service] "prefill-overhead": 0, [e2e-llm-inference-service] "prefill-time-per-token": 0, [e2e-llm-inference-service] "prefill-time-std-dev": 0, [e2e-llm-inference-service] "seed": 1782943406796247600, [e2e-llm-inference-service] "self-signed-certs": false, [e2e-llm-inference-service] "served-model-name": [ [e2e-llm-inference-service] "facebook/opt-125m" [e2e-llm-inference-service] ], [e2e-llm-inference-service] "ssl-certfile": "/var/run/kserve/tls/tls.crt", [e2e-llm-inference-service] "ssl-keyfile": "/var/run/kserve/tls/tls.key", [e2e-llm-inference-service] "time-factor-under-load": 1, [e2e-llm-inference-service] "time-to-first-token": 0, [e2e-llm-inference-service] "time-to-first-token-std-dev": 0, [e2e-llm-inference-service] "tool-call-not-required-param-probability": 50, [e2e-llm-inference-service] "uds-socket-path": "/tmp/tokenizer/tokenizer-uds.socket", [e2e-llm-inference-service] "zmq-endpoint": "tcp://127.0.0.1:5557" [e2e-llm-inference-service] } [e2e-llm-inference-service] > [e2e-llm-inference-service] I0701 22:03:26.832937 1 tokenizer.go:104] "Model is not a real HF model, using simulated tokenizer" model="facebook/opt-125m" [e2e-llm-inference-service] I0701 22:03:26.837182 1 context.go:138] "No dataset path or URL provided, using random text for responses" [e2e-llm-inference-service] I0701 22:03:26.837247 1 communication.go:49] "Starting communication layer" [e2e-llm-inference-service] I0701 22:03:26.837248 1 simulator.go:188] "Start processing routine" [e2e-llm-inference-service] I0701 22:03:26.837496 1 http_server_tls.go:44] "HTTPS server starting with certificate files" cert="/var/run/kserve/tls/tls.crt" key="/var/run/kserve/tls/tls.key" [e2e-llm-inference-service] I0701 22:03:26.837511 1 grpc.go:126] "Server starting" protocol="gRPC" port=8000 [e2e-llm-inference-service] I0701 22:03:26.838126 1 http.go:96] "Server starting" protocol="HTTPS" port=8000 [e2e-llm-inference-service] I0701 22:05:08.784872 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:33870\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:05:12.805528 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:45230\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:05:20.825070 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:45232\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:05:36.847526 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:35038\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:06:08.871540 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:34640\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:07:12.921717 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:58782\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:09:12.978701 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:42684\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:23:22.661574 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:33626\": remote error: tls: bad certificate" [e2e-llm-inference-service] I0701 22:25:22.720314 1 communication.go:105] "Server error" msg="error when serving connection \"10.133.0.38:8000\"<->\"10.134.0.69:49718\": remote error: tls: bad certificate" [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:148 ### Pod scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k (phase=Running) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:188 #### container 'main' (restarts=0) [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:201 # -- logs (current) -- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:diagnostic.py:202 {"level":"info","ts":1782943407.151165,"logger":"setup","caller":"runner/runner.go:196","msg":"GIE build","commit-sha":"181aa8358916e19b8844ccc752b2d6153d4b2ad6","build-ref":"v0.9.0-rc.2"} [e2e-llm-inference-service] Flag --model-server-metrics-scheme has been deprecated, This flag is deprecated. Configure via EndpointPickerConfig data layer plugin parameters instead. [e2e-llm-inference-service] {"level":"info","ts":1782943407.151496,"logger":"setup","caller":"runner/runner.go:217","msg":"Flags processed","flags":{"cert-path":"/var/run/kserve/tls","config-file":"","config-text":"apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\nplugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n","disable-endpoint-subset-filter":false,"enable-cert-reload":true,"enable-grpc-stream-metrics":false,"enable-pprof":true,"endpoint-selector":"","endpoint-target-ports":{},"grpc-health-port":9003,"grpc-max-recv-msg-size":"","grpc-max-send-msg-size":"","grpc-port":9002,"ha-enable-leader-election":false,"health-checking":false,"metrics-endpoint-auth":true,"metrics-port":9090,"metrics-staleness-threshold":2000000000,"model-server-metrics-https-insecure-skip-verify":true,"model-server-metrics-path":"/metrics","model-server-metrics-port":0,"model-server-metrics-scheme":"https","pool-group":"inference.networking.k8s.io","pool-name":"scheduler-v06-pd-migration-test-inference-pool","pool-namespace":"kserve-ci-e2e-test","refresh-metrics-interval":50000000,"refresh-prometheus-metrics-interval":5000000000,"secure-serving":true,"tracing":true,"v":2,"zap-devel":{},"zap-encoder":{},"zap-log-level":{},"zap-stacktrace-level":{},"zap-time-encoding":{}}} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1516318,"logger":"setup.trace","caller":"tracing/telemetry.go:123","msg":"init OTel trace exporter","type":"console"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1524034,"caller":"loader/configloader.go:89","msg":"DEPRECATION: apiVersion inference.networking.x-k8s.io/v1alpha1/EndpointPickerConfig is deprecated","replacement":"llm-d.ai/v1alpha1/EndpointPickerConfig"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1524618,"caller":"loader/configloader.go:121","msg":"Loaded raw configuration","config":"{Plugins: [{Type: disagg-headers-handler} {Type: prefill-filter} {Type: decode-filter} {Type: always-disagg-pd-decider} {Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Type: prefix-cache-scorer} {Type: queue-scorer} {Type: max-score-picker}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}]}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1524777,"logger":"setup","caller":"runner/runner.go:622","msg":"Data layer: ENABLED"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.152748,"logger":"setup","caller":"runner/runner.go:281","msg":"Raw config after phase one","config":{"apiVersion":"inference.networking.x-k8s.io/v1alpha1","dataLayer":null,"kind":"EndpointPickerConfig","plugins":[{"name":"disagg-headers-handler","parameters":null,"type":"disagg-headers-handler"},{"name":"prefill-filter","parameters":null,"type":"prefill-filter"},{"name":"decode-filter","parameters":null,"type":"decode-filter"},{"name":"always-disagg-pd-decider","parameters":null,"type":"always-disagg-pd-decider"},{"name":"disagg-profile-handler","parameters":{"deciders":{"prefill":"always-disagg-pd-decider"}},"type":"disagg-profile-handler"},{"name":"prefix-cache-scorer","parameters":null,"type":"prefix-cache-scorer"},{"name":"queue-scorer","parameters":null,"type":"queue-scorer"},{"name":"max-score-picker","parameters":null,"type":"max-score-picker"}],"schedulingProfiles":[{"name":"prefill","plugins":[{"pluginRef":"prefill-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]},{"name":"decode","plugins":[{"pluginRef":"decode-filter","weight":null},{"pluginRef":"queue-scorer","weight":2},{"pluginRef":"prefix-cache-scorer","weight":3},{"pluginRef":"max-score-picker","weight":null}]}]}} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1691716,"caller":"disagg/disagg_profile_handler.go:186","msg":"No deciders.encode configured, E disaggregation disabled"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1692197,"logger":"utilization-detector/utilization-detector","caller":"utilization/detector.go:83","msg":"Creating new UtilizationDetector","queueDepthThreshold":5,"kvCacheUtilThreshold":0.8,"metricsStalenessThreshold":"200ms","headroom":0} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1693134,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"vllm","mapping":"Mapping{all specs enabled}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1693468,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"sglang","mapping":"Mapping{disabled: [lora]}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1693845,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"trtllm-serve","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1694434,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton-tensorrt-llm","mapping":"Mapping{disabled: [lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1694636,"caller":"metrics/factories.go:230","msg":"Registered engine mapping","engine":"triton","mapping":"Mapping{disabled: [kv, lora, cacheInfo]}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.169514,"caller":"loader/configloader.go:154","msg":"Instantiated all plugins and applied system defaults. Effective raw configuration","config":"{Plugins: [{Name: disagg-headers-handler, Type: disagg-headers-handler} {Name: prefill-filter, Type: prefill-filter} {Name: decode-filter, Type: decode-filter} {Name: always-disagg-pd-decider, Type: always-disagg-pd-decider} {Name: disagg-profile-handler, Type: disagg-profile-handler, Parameters: {\"deciders\":{\"prefill\":\"always-disagg-pd-decider\"}}} {Name: prefix-cache-scorer, Type: prefix-cache-scorer} {Name: queue-scorer, Type: queue-scorer} {Name: max-score-picker, Type: max-score-picker} {Name: fcfs-ordering-policy, Type: fcfs-ordering-policy} {Name: global-strict-fairness-policy, Type: global-strict-fairness-policy} {Name: static-usage-limit-policy, Type: static-usage-limit-policy} {Name: openai-parser, Type: openai-parser} {Name: anthropic-parser, Type: anthropic-parser} {Name: vllmhttp-parser, Type: vllmhttp-parser} {Name: utilization-detector, Type: utilization-detector} {Name: metrics-data-source, Type: metrics-data-source} {Name: core-metrics-extractor, Type: core-metrics-extractor}], SchedulingProfiles: [{Name: prefill, Plugins: [{PluginRef: prefill-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]} {Name: decode, Plugins: [{PluginRef: decode-filter} {PluginRef: queue-scorer, Weight: 2.00} {PluginRef: prefix-cache-scorer, Weight: 3.00} {PluginRef: max-score-picker}]}], DataLayer: {Sources: [{PluginRef: metrics-data-source, Extractors: [{PluginRef: core-metrics-extractor}]}], Discovery: }, FlowControl: {MaxBytes: unlimited, MaxRequests: unlimited, SaturationDetector: {PluginRef: utilization-detector}}, RequestHandler: {Parsers: [{PluginRef: openai-parser}, {PluginRef: anthropic-parser}, {PluginRef: vllmhttp-parser}]}}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1695778,"caller":"approximateprefix/plugin.go:88","msg":"Prefix DataProducer initialized","config":{"autoTune":true,"blockSizeTokens":16,"blockSize":0,"maxPrefixBlocksToMatch":2048,"maxPrefixTokensToMatch":131072,"lruCapacityPerServer":31250}} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1696322,"caller":"approximateprefix/plugin.go:111","msg":"WARNING: configured blockSizeTokens is below the recommended minimum, overriding it.","blockSizeTokens":16,"minimum":64,"issue":"https://github.com/llm-d/llm-d-router/issues/1158"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1696494,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"approx-prefix-cache-producer/approx-prefix-cache-producer","dataKey":"PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer","consumer":"prefix-cache-scorer"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1696668,"caller":"datalayer/data_graph.go:116","msg":"auto-created default producer","producer":"token-producer/token-producer","dataKey":"TokenizedPrompt/token-producer","consumer":"disagg-profile-handler"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.169795,"caller":"runner/runner.go:685","msg":"loaded configuration from file/text successfully"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1698055,"logger":"setup","caller":"runner/runner.go:308","msg":"EPP config after phase two","config":"{SchedulerConfig:{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]} SaturationDetector:0xc000895b80 DataConfig:{Sources:[{Plugin:0xc000152990 Extractors:[0xc000895d80]}]} FlowControlConfig: ParserRegistry:0xc0006ee540}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1846955,"logger":"setup","caller":"runner/runner.go:352","msg":"Setting pprof handlers"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847224,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/symbol"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847355,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/heap"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847405,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/goroutine"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.184745,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/allocs"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847496,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/threadcreate"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.184754,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847582,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/cmdline"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847627,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/profile"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847677,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/trace"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.184772,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/block"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847765,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/pprof/mutex"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1847875,"caller":"manager/internal.go:201","msg":"Registering metrics http server extra handler","path":"/debug/plugins/state"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.184793,"logger":"setup","caller":"runner/runner.go:373","msg":"parsed config","scheduler-config":"{ProfileHandler: disagg-profile-handler/disagg-profile-handler, Profiles: map[decode:{Filters: [decode-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker} prefill:{Filters: [prefill-filter/by-label], Scorers: [queue-scorer/queue-scorer: 2.000000, prefix-cache-scorer/prefix-cache-scorer: 3.000000], Picker: max-score-picker/max-score-picker}]}"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1848204,"logger":"setup","caller":"datalayer/runtime.go:99","msg":"Configuring datalayer runtime","numSources":1} [e2e-llm-inference-service] {"level":"info","ts":1782943407.184832,"logger":"setup","caller":"datalayer/runtime.go:118","msg":"Processing source","source":"metrics-data-source","numExtractors":1} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1848469,"logger":"setup","caller":"datalayer/runtime.go:147","msg":"Source configured","source":"metrics-data-source","extractors":["core-metrics-extractor/core-metrics-extractor"]} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1848783,"logger":"setup","caller":"datalayer/runtime.go:206","msg":"Datalayer runtime configured","pollers":1,"notifiers":0,"endpointSources":0} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1848874,"logger":"setup","caller":"runner/runner.go:833","msg":"Experimental Flow Control layer is disabled, using legacy admission control"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1849747,"logger":"setup","caller":"runner/runner.go:721","msg":"ExtProc server runner added to manager."} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1849859,"logger":"setup","caller":"runner/runner.go:260","msg":"Controller manager starting"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1850116,"logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.185246,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"health"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1853163,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"health","port":9003} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1854548,"logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":9090","secure":false} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1854868,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","source":"kind source: *v1.InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1855397,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","source":"kind source: *v1alpha2.InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.185575,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","source":"kind source: *v1alpha2.InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1856036,"caller":"controller/controller.go:370","msg":"Starting EventSource","controller":"pod","controllerGroup":"","controllerKind":"Pod","source":"kind source: *v1.Pod"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1862876,"caller":"runnable/grpc.go:35","msg":"gRPC server starting","name":"ext-proc"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.186355,"caller":"runnable/grpc.go:43","msg":"gRPC server listening","name":"ext-proc","port":9002} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1890326,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceModelRewrite","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.1890593,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1alpha2.InferenceObjective","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.189378,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.InferencePool","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.190335,"logger":"controller-runtime.cache","caller":"cache/reflector.go:446","msg":"Caches populated","type":"*v1.Pod","reflector":"pkg/mod/k8s.io/client-go@v0.35.6/tools/cache/reflector.go:289"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.286924,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.2870014,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencemodelrewrite","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceModelRewrite","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782943407.2870677,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.2870975,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferenceobjective","controllerGroup":"inference.networking.x-k8s.io","controllerKind":"InferenceObjective","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782943407.2871463,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.2871802,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782943407.2873385,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"scheduler-v06-pd-migration-test-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"scheduler-v06-pd-migration-test-inference-pool","reconcileID":"d2f4383c-a839-4020-9a37-a150b299b3e9"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.3871105,"caller":"controller/controller.go:303","msg":"Starting Controller","controller":"pod","controllerGroup":"","controllerKind":"Pod"} [e2e-llm-inference-service] {"level":"info","ts":1782943407.387161,"caller":"controller/controller.go:306","msg":"Starting workers","controller":"pod","controllerGroup":"","controllerKind":"Pod","worker count":1} [e2e-llm-inference-service] {"level":"info","ts":1782943415.0283349,"caller":"controller/inferencepool_reconciler.go:46","msg":"Reconciling InferencePool","controller":"inferencepool","controllerGroup":"inference.networking.k8s.io","controllerKind":"InferencePool","InferencePool":{"name":"scheduler-v06-pd-migration-test-inference-pool","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"scheduler-v06-pd-migration-test-inference-pool","reconcileID":"cb88da75-72d6-4f75-a1b8-f3f1a9e340d1"} [e2e-llm-inference-service] {"level":"info","ts":1782943416.8656404,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk","reconcileID":"48df46fb-e37a-4227-bfbe-76acece97fbf"} [e2e-llm-inference-service] {"level":"info","ts":1782943418.03071,"caller":"controller/pod_reconciler.go:99","msg":"Pod already exists","controller":"pod","controllerGroup":"","controllerKind":"Pod","Pod":{"name":"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz","namespace":"kserve-ci-e2e-test"},"namespace":"kserve-ci-e2e-test","name":"scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz","reconcileID":"c8a0f66e-1a67-4617-8711-112b04684ed2"} [e2e-llm-inference-service] {"level":"info","ts":1782943508.756032,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"b36871c0-98a6-4ebd-9223-c711ffd70440"} [e2e-llm-inference-service] {"level":"info","ts":1782943508.7566388,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"b36871c0-98a6-4ebd-9223-c711ffd70440","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782943512.7907004,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"e9af1e96-a8e4-40f2-81f0-eb85191fdeac"} [e2e-llm-inference-service] {"level":"info","ts":1782943512.7910526,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"e9af1e96-a8e4-40f2-81f0-eb85191fdeac","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782943520.8101919,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"d50fc53b-91d7-4509-8306-6e537ac6b73c"} [e2e-llm-inference-service] {"level":"info","ts":1782943520.8104923,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"d50fc53b-91d7-4509-8306-6e537ac6b73c","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782943536.8318226,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"69b22a95-d717-4a80-8efa-0c9f59bcd670"} [e2e-llm-inference-service] {"level":"info","ts":1782943536.8321395,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"69b22a95-d717-4a80-8efa-0c9f59bcd670","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782943568.8556023,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"ac2db612-ef37-473a-b255-43042d1c28e1"} [e2e-llm-inference-service] {"level":"info","ts":1782943568.8559277,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"ac2db612-ef37-473a-b255-43042d1c28e1","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782943632.90642,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"cc7003ee-42a4-4ce6-8cee-f09a959eb0a5"} [e2e-llm-inference-service] {"level":"info","ts":1782943632.9067023,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"cc7003ee-42a4-4ce6-8cee-f09a959eb0a5","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782943752.9589617,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"ec73e815-5892-4730-a6ef-70643244d7c2"} [e2e-llm-inference-service] {"level":"info","ts":1782943752.9592898,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"ec73e815-5892-4730-a6ef-70643244d7c2","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782944602.6435661,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"10d2adf8-0f03-406c-ad7d-3a2301ab9a86"} [e2e-llm-inference-service] {"level":"info","ts":1782944602.643836,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"10d2adf8-0f03-406c-ad7d-3a2301ab9a86","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] {"level":"info","ts":1782944722.701481,"caller":"handlers/server.go:395","msg":"EPP received request","x-request-id":"05d92e72-1936-4092-989d-9505e0b15924"} [e2e-llm-inference-service] {"level":"info","ts":1782944722.7017863,"caller":"handlers/server.go:656","msg":"EPP sent request body response(s) to proxy","x-request-id":"05d92e72-1936-4092-989d-9505e0b15924","modelName":"facebook/opt-125m","targetModelName":"facebook/opt-125m"} [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 66a34643-95bb-42c2-8dd7-fc6de8206409 [e2e-llm-inference-service] resourceVersion: '82087' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.132.0.72 [e2e-llm-inference-service] nodeName: ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k [e2e-llm-inference-service] uid: b1874861-4e64-470f-8964-44c6e8fd5683 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: abbe1c4e-0a94-42cf-b682-83dfcbd39bad [e2e-llm-inference-service] resourceVersion: '81847' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpoints.kubernetes.io/managed-by: endpoint-controller [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:subsets: {} [e2e-llm-inference-service] subsets: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - ip: 10.133.0.38 [e2e-llm-inference-service] nodeName: ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk [e2e-llm-inference-service] uid: 44254112-edab-4268-8b4b-9c43cb7e371b [e2e-llm-inference-service] - ip: 10.134.0.69 [e2e-llm-inference-service] nodeName: ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz [e2e-llm-inference-service] uid: 70d7b504-f4b5-4838-8df9-c3629355aa72 [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Endpoints [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-5c87756974- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 70d7b504-f4b5-4838-8df9-c3629355aa72 [e2e-llm-inference-service] resourceVersion: '81843' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.134.0.69/23"],"mac_address":"0a:58:0a:86:00:45","gateway_ips":["10.134.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.134.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.134.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.134.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.134.0.1"}],"ip_address":"10.134.0.69/23","gateway_ip":"10.134.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.134.0.69\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:86:00:45\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974 [e2e-llm-inference-service] uid: fc9762ae-c2a9-4761-892f-6e9d9be95307 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-132-117 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"fc9762ae-c2a9-4761-892f-6e9d9be95307"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:initContainerStatuses: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.134.0.69"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-czsgj [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-czsgj [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-czsgj [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] nodeName: ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-srv9x [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] hostIP: 10.0.132.117 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.132.117 [e2e-llm-inference-service] podIP: 10.134.0.69 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.134.0.69 [e2e-llm-inference-service] startTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] initContainerStatuses: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-disagg-sidecar@sha256:1ec2c035216ea0a4f8a6f7ccda55398fc3d9f71f487c135f49ba0dea08163f12 [e2e-llm-inference-service] containerID: cri-o://cba610fa38e690fecb0ab799e0eb2e77a105ef84a0e05f86340a190db1ececc1 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-czsgj [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://70e6b3d82670e8c0f0d049bad6cc8c6b2f0ca01ea609c5499a00df7cba72e511 [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-czsgj [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 44254112-edab-4268-8b4b-9c43cb7e371b [e2e-llm-inference-service] resourceVersion: '81822' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.133.0.38/23"],"mac_address":"0a:58:0a:85:00:26","gateway_ips":["10.133.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.133.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.133.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.133.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.133.0.1"}],"ip_address":"10.133.0.38/23","gateway_ip":"10.133.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.133.0.38\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:85:00:26\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 [e2e-llm-inference-service] uid: e828a000-0844-4d14-a0c5-aa15eaffa1ff [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-140-104 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"e828a000-0844-4d14-a0c5-aa15eaffa1ff"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.133.0.38"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: kube-api-access-cn9x2 [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: kube-api-access-cn9x2 [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: default [e2e-llm-inference-service] serviceAccount: default [e2e-llm-inference-service] nodeName: ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] hostIP: 10.0.140.104 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.140.104 [e2e-llm-inference-service] podIP: 10.133.0.38 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.133.0.38 [e2e-llm-inference-service] startTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-inference-sim@sha256:bab162bd25e2ed8b15022387cdb223023aeb33be49476af9f0115c0398fb8ff5 [e2e-llm-inference-service] containerID: cri-o://1feff14011ec6b13646ec3319f35411f7f7d0c939c4a5edacb16159c214a43dc [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: kube-api-access-cn9x2 [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: b1874861-4e64-470f-8964-44c6e8fd5683 [e2e-llm-inference-service] resourceVersion: '82085' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.132.0.72/23"],"mac_address":"0a:58:0a:84:00:48","gateway_ips":["10.132.0.1"],"routes":[{"dest":"10.132.0.0/14","nextHop":"10.132.0.1"},{"dest":"172.31.0.0/16","nextHop":"10.132.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.132.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.132.0.1"}],"ip_address":"10.132.0.72/23","gateway_ip":"10.132.0.1","role":"primary"}}' [e2e-llm-inference-service] k8s.v1.cni.cncf.io/network-status: "[{\n \"name\": \"ovn-kubernetes\",\n \ [e2e-llm-inference-service] \ \"interface\": \"eth0\",\n \"ips\": [\n \"10.132.0.72\"\n ],\n\ [e2e-llm-inference-service] \ \"mac\": \"0a:58:0a:84:00:48\",\n \"default\": true,\n \"dns\": {}\n\ [e2e-llm-inference-service] }]" [e2e-llm-inference-service] openshift.io/scc: restricted-v2 [e2e-llm-inference-service] seccomp.security.alpha.kubernetes.io/pod: runtime/default [e2e-llm-inference-service] security.openshift.io/validated-scc-subject-type: user [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd [e2e-llm-inference-service] uid: 1633a22d-edf9-4657-b45d-95b9c01ea845 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: ip-10-0-138-219 [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.ovn.org/pod-networks: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"1633a22d-edf9-4657-b45d-95b9c01ea845"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:enableServiceLinks: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: multus-daemon [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:k8s.v1.cni.cncf.io/network-status: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] - manager: kubelet [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] k:{"type":"ContainersReady"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Initialized"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodReadyToStartContainers"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"PodScheduled"}: [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] k:{"type":"Ready"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastProbeTime: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:containerStatuses: {} [e2e-llm-inference-service] f:hostIP: {} [e2e-llm-inference-service] f:hostIPs: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:phase: {} [e2e-llm-inference-service] f:podIP: {} [e2e-llm-inference-service] f:podIPs: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"ip":"10.132.0.72"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:ip: {} [e2e-llm-inference-service] f:startTime: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: kube-api-access-gcdsh [e2e-llm-inference-service] projected: [e2e-llm-inference-service] sources: [e2e-llm-inference-service] - serviceAccountToken: [e2e-llm-inference-service] expirationSeconds: 3607 [e2e-llm-inference-service] path: token [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: kube-root-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: ca.crt [e2e-llm-inference-service] path: ca.crt [e2e-llm-inference-service] - downwardAPI: [e2e-llm-inference-service] items: [e2e-llm-inference-service] - path: namespace [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - configMap: [e2e-llm-inference-service] name: openshift-service-ca.crt [e2e-llm-inference-service] items: [e2e-llm-inference-service] - key: service-ca.crt [e2e-llm-inference-service] path: service-ca.crt [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type: decode-filter\n\ [e2e-llm-inference-service] - type: always-disagg-pd-decider\n- parameters:\n deciders:\n prefill:\ [e2e-llm-inference-service] \ always-disagg-pd-decider\n type: disagg-profile-handler\n- type: prefix-cache-scorer\n\ [e2e-llm-inference-service] - type: queue-scorer\n- type: max-score-picker\nschedulingProfiles:\n- name:\ [e2e-llm-inference-service] \ prefill\n plugins:\n - pluginRef: prefill-filter\n - pluginRef: queue-scorer\n\ [e2e-llm-inference-service] \ weight: 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef:\ [e2e-llm-inference-service] \ max-score-picker\n- name: decode\n plugins:\n - pluginRef: decode-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-gcdsh [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsUser: 1000690000 [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] nodeName: ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] seLinuxOptions: [e2e-llm-inference-service] level: s0:c26,c20 [e2e-llm-inference-service] fsGroup: 1000690000 [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-24ndg [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] tolerations: [e2e-llm-inference-service] - key: node.kubernetes.io/not-ready [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/unreachable [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoExecute [e2e-llm-inference-service] tolerationSeconds: 300 [e2e-llm-inference-service] - key: node.kubernetes.io/memory-pressure [e2e-llm-inference-service] operator: Exists [e2e-llm-inference-service] effect: NoSchedule [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] enableServiceLinks: true [e2e-llm-inference-service] preemptionPolicy: PreemptLowerPriority [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] phase: Running [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: PodReadyToStartContainers [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] - type: Initialized [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] - type: Ready [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] - type: ContainersReady [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] - type: PodScheduled [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastProbeTime: null [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] hostIP: 10.0.138.219 [e2e-llm-inference-service] hostIPs: [e2e-llm-inference-service] - ip: 10.0.138.219 [e2e-llm-inference-service] podIP: 10.132.0.72 [e2e-llm-inference-service] podIPs: [e2e-llm-inference-service] - ip: 10.132.0.72 [e2e-llm-inference-service] startTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] containerStatuses: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] state: [e2e-llm-inference-service] running: [e2e-llm-inference-service] startedAt: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] lastState: {} [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] restartCount: 0 [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] imageID: ghcr.io/llm-d/llm-d-router-endpoint-picker@sha256:06b6c75d77afd0e07053402752a9736c2dfbc12a306d0d37d963aac4c1d4e6a6 [e2e-llm-inference-service] containerID: cri-o://3fc52102dbec9138800878784ebdab0d26834bc02750b36350ac9e89cef8960c [e2e-llm-inference-service] started: true [e2e-llm-inference-service] allocatedResources: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] - name: kube-api-access-gcdsh [e2e-llm-inference-service] mountPath: /var/run/secrets/kubernetes.io/serviceaccount [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] recursiveReadOnly: Disabled [e2e-llm-inference-service] user: [e2e-llm-inference-service] linux: [e2e-llm-inference-service] uid: 1000690000 [e2e-llm-inference-service] gid: 0 [e2e-llm-inference-service] supplementalGroups: [e2e-llm-inference-service] - 0 [e2e-llm-inference-service] - 1000690000 [e2e-llm-inference-service] qosClass: Burstable [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 64ed2fec-4af0-4123-9bc0-805031d16ba6 [e2e-llm-inference-service] resourceVersion: '81500' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-epp-sa-dockercfg-24ndg [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-epp-sa-dockercfg-24ndg"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-76tz7"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-24ndg [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-epp-sa-dockercfg-24ndg [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: acb9e527-05a3-4a71-b70d-a65ac38b3c2c [e2e-llm-inference-service] resourceVersion: '81452' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] openshift.io/internal-registry-pull-secret-ref: scheduler-v06-pd-migration-test-kserve-dockercfg-srv9x [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: openshift.io/image-registry-pull-secrets_service-account-controller [e2e-llm-inference-service] operation: Apply [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:imagePullSecrets: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] f:openshift.io/internal-registry-pull-secret-ref: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] k:{"name":"scheduler-v06-pd-migration-test-kserve-dockercfg-srv9x"}: {} [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:secrets: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"default-dockercfg-76tz7"}: {} [e2e-llm-inference-service] k:{"name":"seaweedfs-s3-creds"}: {} [e2e-llm-inference-service] secrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] - name: seaweedfs-s3-creds [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-srv9x [e2e-llm-inference-service] imagePullSecrets: [e2e-llm-inference-service] - name: default-dockercfg-76tz7 [e2e-llm-inference-service] - name: scheduler-v06-pd-migration-test-kserve-dockercfg-srv9x [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: ServiceAccount [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 5b2a57ed-e783-42e7-bf52-dfcbeecb0ac5 [e2e-llm-inference-service] resourceVersion: '81517' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] k:{"port":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] targetPort: grpc [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] targetPort: grpc-health [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] targetPort: metrics [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] targetPort: zmq [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] clusterIP: 172.31.252.51 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.252.51 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 0e12f54b-39cf-434e-a45e-ff6e09c64ecf [e2e-llm-inference-service] resourceVersion: '81473' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:internalTrafficPolicy: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"port":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:targetPort: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:sessionAffinity: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] spec: [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] targetPort: 8000 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] clusterIP: 172.31.11.39 [e2e-llm-inference-service] clusterIPs: [e2e-llm-inference-service] - 172.31.11.39 [e2e-llm-inference-service] type: ClusterIP [e2e-llm-inference-service] sessionAffinity: None [e2e-llm-inference-service] ipFamilies: [e2e-llm-inference-service] - IPv4 [e2e-llm-inference-service] ipFamilyPolicy: SingleStack [e2e-llm-inference-service] internalTrafficPolicy: Cluster [e2e-llm-inference-service] status: [e2e-llm-inference-service] loadBalancer: {} [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 55109cbc-e00f-4f3b-8082-f9b8319501fd [e2e-llm-inference-service] resourceVersion: '81849' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-5c87756974" has successfully [e2e-llm-inference-service] progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: edaa0147-87a5-4f5c-b294-f1fc2b34a7ef [e2e-llm-inference-service] resourceVersion: '81828' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:rollingUpdate: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:maxSurge: {} [e2e-llm-inference-service] f:maxUnavailable: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: RollingUpdate [e2e-llm-inference-service] rollingUpdate: [e2e-llm-inference-service] maxUnavailable: 25% [e2e-llm-inference-service] maxSurge: 25% [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 7e82c7d7-5e3e-47cd-b049-57f9e0ea9ed7 [e2e-llm-inference-service] resourceVersion: '82089' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:progressDeadlineSeconds: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:revisionHistoryLimit: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:strategy: [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Available"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Progressing"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:lastUpdateTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:updatedReplicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n\ [e2e-llm-inference-service] \ prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] strategy: [e2e-llm-inference-service] type: Recreate [e2e-llm-inference-service] revisionHistoryLimit: 10 [e2e-llm-inference-service] progressDeadlineSeconds: 600 [e2e-llm-inference-service] status: [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] updatedReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - type: Available [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] reason: MinimumReplicasAvailable [e2e-llm-inference-service] message: Deployment has minimum availability. [e2e-llm-inference-service] - type: Progressing [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] lastUpdateTime: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] reason: NewReplicaSetAvailable [e2e-llm-inference-service] message: ReplicaSet "scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd" [e2e-llm-inference-service] has successfully progressed. [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: fc9762ae-c2a9-4761-892f-6e9d9be95307 [e2e-llm-inference-service] resourceVersion: '81848' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] uid: 55109cbc-e00f-4f3b-8082-f9b8319501fd [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"55109cbc-e00f-4f3b-8082-f9b8319501fd"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8001,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:initContainers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"llm-d-routing-sidecar"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"INFERENCE_POOL_NAMESPACE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:valueFrom: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fieldRef: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] initContainers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-disagg-sidecar:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/pd-sidecar [e2e-llm-inference-service] - --port=8000 [e2e-llm-inference-service] - --vllm-port=8001 [e2e-llm-inference-service] - --kv-connector=nixlv2 [e2e-llm-inference-service] - --enable-ssrf-protection=true [e2e-llm-inference-service] - --pool-group=inference.networking.x-k8s.io [e2e-llm-inference-service] - --inference-pool=kserve-ci-e2e-test/scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --secure-proxy=true [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] - --decoder-use-tls=true [e2e-llm-inference-service] - --prefiller-use-tls=true [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: INFERENCE_POOL_NAMESPACE [e2e-llm-inference-service] valueFrom: [e2e-llm-inference-service] fieldRef: [e2e-llm-inference-service] apiVersion: v1 [e2e-llm-inference-service] fieldPath: metadata.namespace [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] - name: INFERENCE_POOL_NAME [e2e-llm-inference-service] value: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] resources: {} [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 10 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] initialDelaySeconds: 10 [e2e-llm-inference-service] timeoutSeconds: 5 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8001' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8001 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8001 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847 [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: e828a000-0844-4d14-a0c5-aa15eaffa1ff [e2e-llm-inference-service] resourceVersion: '81827' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '2' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill [e2e-llm-inference-service] uid: edaa0147-87a5-4f5c-b294-f1fc2b34a7ef [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"edaa0147-87a5-4f5c-b294-f1fc2b34a7ef"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:llm-d.ai/role: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"HF_HUB_CACHE"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"HOME"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] k:{"name":"VLLM_LOGGING_LEVEL"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":8000,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:limits: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:startupProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:httpGet: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:path: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:scheme: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/dev/shm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/models"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"dshm"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:medium: {} [e2e-llm-inference-service] f:sizeLimit: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"home"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"model-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tmp-dir"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:36Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] emptyDir: [e2e-llm-inference-service] medium: Memory [e2e-llm-inference-service] sizeLimit: 1Gi [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-inference-sim:v0.8.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/llm-d-inference-sim [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --port [e2e-llm-inference-service] - '8000' [e2e-llm-inference-service] - --model [e2e-llm-inference-service] - facebook/opt-125m [e2e-llm-inference-service] - --mode [e2e-llm-inference-service] - random [e2e-llm-inference-service] - --ssl-certfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.crt [e2e-llm-inference-service] - --ssl-keyfile [e2e-llm-inference-service] - /var/run/kserve/tls/tls.key [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - containerPort: 8000 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: HOME [e2e-llm-inference-service] value: /home [e2e-llm-inference-service] - name: VLLM_LOGGING_LEVEL [e2e-llm-inference-service] value: INFO [e2e-llm-inference-service] - name: HF_HUB_CACHE [e2e-llm-inference-service] value: /models [e2e-llm-inference-service] resources: [e2e-llm-inference-service] limits: [e2e-llm-inference-service] cpu: '1' [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 200m [e2e-llm-inference-service] memory: 2Gi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: home [e2e-llm-inference-service] mountPath: /home [e2e-llm-inference-service] - name: tmp-dir [e2e-llm-inference-service] mountPath: /tmp [e2e-llm-inference-service] - name: dshm [e2e-llm-inference-service] mountPath: /dev/shm [e2e-llm-inference-service] - name: model-cache [e2e-llm-inference-service] mountPath: /models [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 10 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 1 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 2 [e2e-llm-inference-service] startupProbe: [e2e-llm-inference-service] httpGet: [e2e-llm-inference-service] path: /health [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] scheme: HTTPS [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 60 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: File [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59dc679cd [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1633a22d-edf9-4657-b45d-95b9c01ea845 [e2e-llm-inference-service] resourceVersion: '82088' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] deployment.kubernetes.io/desired-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/max-replicas: '1' [e2e-llm-inference-service] deployment.kubernetes.io/revision: '1' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: apps/v1 [e2e-llm-inference-service] kind: Deployment [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler [e2e-llm-inference-service] uid: 7e82c7d7-5e3e-47cd-b049-57f9e0ea9ed7 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/desired-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/max-replicas: {} [e2e-llm-inference-service] f:deployment.kubernetes.io/revision: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"7e82c7d7-5e3e-47cd-b049-57f9e0ea9ed7"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] f:selector: {} [e2e-llm-inference-service] f:template: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/version: {} [e2e-llm-inference-service] f:certificates.kserve.io/expiration-v2: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:pod-template-hash: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] f:containers: [e2e-llm-inference-service] k:{"name":"main"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:args: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:env: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"SSL_CERT_DIR"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:image: {} [e2e-llm-inference-service] f:imagePullPolicy: {} [e2e-llm-inference-service] f:lifecycle: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:preStop: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:command: {} [e2e-llm-inference-service] f:livenessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:ports: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"containerPort":5557,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9002,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9003,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] k:{"containerPort":9090,"protocol":"TCP"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:containerPort: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:protocol: {} [e2e-llm-inference-service] f:readinessProbe: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureThreshold: {} [e2e-llm-inference-service] f:grpc: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:port: {} [e2e-llm-inference-service] f:service: {} [e2e-llm-inference-service] f:initialDelaySeconds: {} [e2e-llm-inference-service] f:periodSeconds: {} [e2e-llm-inference-service] f:successThreshold: {} [e2e-llm-inference-service] f:timeoutSeconds: {} [e2e-llm-inference-service] f:resources: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:requests: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:cpu: {} [e2e-llm-inference-service] f:memory: {} [e2e-llm-inference-service] f:securityContext: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:allowPrivilegeEscalation: {} [e2e-llm-inference-service] f:capabilities: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:drop: {} [e2e-llm-inference-service] f:readOnlyRootFilesystem: {} [e2e-llm-inference-service] f:runAsNonRoot: {} [e2e-llm-inference-service] f:seccompProfile: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:terminationMessagePath: {} [e2e-llm-inference-service] f:terminationMessagePolicy: {} [e2e-llm-inference-service] f:volumeMounts: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"mountPath":"/tmp/tokenizer"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"mountPath":"/var/run/kserve/tls"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:mountPath: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:readOnly: {} [e2e-llm-inference-service] f:dnsPolicy: {} [e2e-llm-inference-service] f:restartPolicy: {} [e2e-llm-inference-service] f:schedulerName: {} [e2e-llm-inference-service] f:securityContext: {} [e2e-llm-inference-service] f:serviceAccount: {} [e2e-llm-inference-service] f:serviceAccountName: {} [e2e-llm-inference-service] f:terminationGracePeriodSeconds: {} [e2e-llm-inference-service] f:volumes: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"name":"tls-certs"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:secret: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:defaultMode: {} [e2e-llm-inference-service] f:secretName: {} [e2e-llm-inference-service] k:{"name":"tokenizer-cache"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-tmp"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] k:{"name":"tokenizer-uds"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:emptyDir: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:availableReplicas: {} [e2e-llm-inference-service] f:fullyLabeledReplicas: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] f:readyReplicas: {} [e2e-llm-inference-service] f:replicas: {} [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] spec: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] template: [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] app.kubernetes.io/version: 0.9.0 [e2e-llm-inference-service] certificates.kserve.io/expiration-v2: 'true' [e2e-llm-inference-service] spec: [e2e-llm-inference-service] volumes: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] secret: [e2e-llm-inference-service] secretName: scheduler-v06-pd-migration-test-kserve-self-signed-certs [e2e-llm-inference-service] defaultMode: 420 [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-tmp [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] - name: tokenizer-cache [e2e-llm-inference-service] emptyDir: {} [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.9.0-rc.2 [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /app/epp [e2e-llm-inference-service] - --pool-name [e2e-llm-inference-service] - scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] - --pool-namespace [e2e-llm-inference-service] - kserve-ci-e2e-test [e2e-llm-inference-service] - --zap-encoder [e2e-llm-inference-service] - json [e2e-llm-inference-service] - --grpc-port [e2e-llm-inference-service] - '9002' [e2e-llm-inference-service] - --grpc-health-port [e2e-llm-inference-service] - '9003' [e2e-llm-inference-service] - --enable-cert-reload=true [e2e-llm-inference-service] - --secure-serving=true [e2e-llm-inference-service] - --model-server-metrics-scheme=https [e2e-llm-inference-service] - --cert-path=/var/run/kserve/tls [e2e-llm-inference-service] args: [e2e-llm-inference-service] - --config-text [e2e-llm-inference-service] - "apiVersion: inference.networking.x-k8s.io/v1alpha1\nkind: EndpointPickerConfig\n\ [e2e-llm-inference-service] plugins:\n- type: disagg-headers-handler\n- type: prefill-filter\n- type:\ [e2e-llm-inference-service] \ decode-filter\n- type: always-disagg-pd-decider\n- parameters:\n deciders:\n\ [e2e-llm-inference-service] \ prefill: always-disagg-pd-decider\n type: disagg-profile-handler\n\ [e2e-llm-inference-service] - type: prefix-cache-scorer\n- type: queue-scorer\n- type: max-score-picker\n\ [e2e-llm-inference-service] schedulingProfiles:\n- name: prefill\n plugins:\n - pluginRef: prefill-filter\n\ [e2e-llm-inference-service] \ - pluginRef: queue-scorer\n weight: 2\n - pluginRef: prefix-cache-scorer\n\ [e2e-llm-inference-service] \ weight: 3\n - pluginRef: max-score-picker\n- name: decode\n plugins:\n\ [e2e-llm-inference-service] \ - pluginRef: decode-filter\n - pluginRef: queue-scorer\n weight:\ [e2e-llm-inference-service] \ 2\n - pluginRef: prefix-cache-scorer\n weight: 3\n - pluginRef: max-score-picker\n" [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] containerPort: 9002 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] containerPort: 9003 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] containerPort: 9090 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] containerPort: 5557 [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] env: [e2e-llm-inference-service] - name: SSL_CERT_DIR [e2e-llm-inference-service] value: /var/run/kserve/tls:/var/run/secrets/kubernetes.io/serviceaccount:/etc/pki/tls/certs [e2e-llm-inference-service] resources: [e2e-llm-inference-service] requests: [e2e-llm-inference-service] cpu: 256m [e2e-llm-inference-service] memory: 500Mi [e2e-llm-inference-service] volumeMounts: [e2e-llm-inference-service] - name: tls-certs [e2e-llm-inference-service] readOnly: true [e2e-llm-inference-service] mountPath: /var/run/kserve/tls [e2e-llm-inference-service] - name: tokenizer-uds [e2e-llm-inference-service] mountPath: /tmp/tokenizer [e2e-llm-inference-service] livenessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: liveness [e2e-llm-inference-service] initialDelaySeconds: 5 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] readinessProbe: [e2e-llm-inference-service] grpc: [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] service: readiness [e2e-llm-inference-service] initialDelaySeconds: 30 [e2e-llm-inference-service] timeoutSeconds: 1 [e2e-llm-inference-service] periodSeconds: 10 [e2e-llm-inference-service] successThreshold: 1 [e2e-llm-inference-service] failureThreshold: 3 [e2e-llm-inference-service] lifecycle: [e2e-llm-inference-service] preStop: [e2e-llm-inference-service] exec: [e2e-llm-inference-service] command: [e2e-llm-inference-service] - /bin/sleep [e2e-llm-inference-service] - '15' [e2e-llm-inference-service] terminationMessagePath: /dev/termination-log [e2e-llm-inference-service] terminationMessagePolicy: FallbackToLogsOnError [e2e-llm-inference-service] imagePullPolicy: IfNotPresent [e2e-llm-inference-service] securityContext: [e2e-llm-inference-service] capabilities: [e2e-llm-inference-service] drop: [e2e-llm-inference-service] - ALL [e2e-llm-inference-service] runAsNonRoot: true [e2e-llm-inference-service] readOnlyRootFilesystem: true [e2e-llm-inference-service] allowPrivilegeEscalation: false [e2e-llm-inference-service] seccompProfile: [e2e-llm-inference-service] type: RuntimeDefault [e2e-llm-inference-service] restartPolicy: Always [e2e-llm-inference-service] terminationGracePeriodSeconds: 60 [e2e-llm-inference-service] dnsPolicy: ClusterFirst [e2e-llm-inference-service] serviceAccountName: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] serviceAccount: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] securityContext: {} [e2e-llm-inference-service] schedulerName: default-scheduler [e2e-llm-inference-service] status: [e2e-llm-inference-service] replicas: 1 [e2e-llm-inference-service] fullyLabeledReplicas: 1 [e2e-llm-inference-service] readyReplicas: 1 [e2e-llm-inference-service] availableReplicas: 1 [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] apiVersion: apps/v1 [e2e-llm-inference-service] kind: ReplicaSet [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 34c8e6ec-9aa6-4f48-a760-948b2cb0067a [e2e-llm-inference-service] resourceVersion: '81508' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4d0a0e4b-35c9-4a0d-9762-504a72b8adb4 [e2e-llm-inference-service] resourceVersion: '81458' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] apiGroup: rbac.authorization.k8s.io [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 59497139-6568-4163-a079-eac22be3a330 [e2e-llm-inference-service] resourceVersion: '81506' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] - create [e2e-llm-inference-service] - update [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - delete [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 96c51f67-cc63-4512-9980-cf57fe5bfaad [e2e-llm-inference-service] resourceVersion: '81455' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service-sckfq [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-epp-service- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 04a3b2cf-8dbc-483b-ae6b-5e37195d6dea [e2e-llm-inference-service] resourceVersion: '82086' [e2e-llm-inference-service] generation: 3 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] uid: 5b2a57ed-e783-42e7-bf52-dfcbeecb0ac5 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:58Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"5b2a57ed-e783-42e7-bf52-dfcbeecb0ac5"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.132.0.72 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k [e2e-llm-inference-service] uid: b1874861-4e64-470f-8964-44c6e8fd5683 [e2e-llm-inference-service] nodeName: ip-10-0-138-219.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: grpc [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9002 [e2e-llm-inference-service] - name: grpc-health [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9003 [e2e-llm-inference-service] - name: metrics [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 9090 [e2e-llm-inference-service] - name: zmq [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 5557 [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc-ps827 [e2e-llm-inference-service] generateName: scheduler-v06-pd-migration-test-kserve-workload-svc- [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 1f8c3591-8a93-43db-a967-0246ded81e99 [e2e-llm-inference-service] resourceVersion: '81846' [e2e-llm-inference-service] generation: 5 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io [e2e-llm-inference-service] kubernetes.io/service-name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] endpoints.kubernetes.io/last-change-trigger-time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: v1 [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] uid: 0e12f54b-39cf-434e-a45e-ff6e09c64ecf [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: kube-controller-manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:38Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:addressType: {} [e2e-llm-inference-service] f:endpoints: {} [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:endpoints.kubernetes.io/last-change-trigger-time: {} [e2e-llm-inference-service] f:generateName: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:endpointslice.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:kubernetes.io/service-name: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"0e12f54b-39cf-434e-a45e-ff6e09c64ecf"}: {} [e2e-llm-inference-service] f:ports: {} [e2e-llm-inference-service] addressType: IPv4 [e2e-llm-inference-service] endpoints: [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.133.0.38 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk [e2e-llm-inference-service] uid: 44254112-edab-4268-8b4b-9c43cb7e371b [e2e-llm-inference-service] nodeName: ip-10-0-140-104.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] - addresses: [e2e-llm-inference-service] - 10.134.0.69 [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] ready: true [e2e-llm-inference-service] serving: true [e2e-llm-inference-service] terminating: false [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] kind: Pod [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz [e2e-llm-inference-service] uid: 70d7b504-f4b5-4838-8df9-c3629355aa72 [e2e-llm-inference-service] nodeName: ip-10-0-132-117.ec2.internal [e2e-llm-inference-service] zone: us-east-1a [e2e-llm-inference-service] ports: [e2e-llm-inference-service] - name: https [e2e-llm-inference-service] protocol: TCP [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] appProtocol: https [e2e-llm-inference-service] apiVersion: discovery.k8s.io/v1 [e2e-llm-inference-service] kind: EndpointSlice [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 34c8e6ec-9aa6-4f48-a760-948b2cb0067a [e2e-llm-inference-service] resourceVersion: '81508' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-sa [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-rb [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 4d0a0e4b-35c9-4a0d-9762-504a72b8adb4 [e2e-llm-inference-service] resourceVersion: '81458' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:roleRef: {} [e2e-llm-inference-service] f:subjects: {} [e2e-llm-inference-service] userNames: [e2e-llm-inference-service] - system:serviceaccount:kserve-ci-e2e-test:scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] groupNames: null [e2e-llm-inference-service] subjects: [e2e-llm-inference-service] - kind: ServiceAccount [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve [e2e-llm-inference-service] roleRef: [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: RoleBinding [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 59497139-6568-4163-a079-eac22be3a330 [e2e-llm-inference-service] resourceVersion: '81506' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodels [e2e-llm-inference-service] - inferenceobjectives [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencemodelrewrites [e2e-llm-inference-service] - inferencepoolimports [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - discovery.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - endpointslices [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - create [e2e-llm-inference-service] - delete [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - patch [e2e-llm-inference-service] - update [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - coordination.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - leases [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-role [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] uid: 96c51f67-cc63-4512-9980-cf57fe5bfaad [e2e-llm-inference-service] resourceVersion: '81455' [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] apiVersion: rbac.authorization.k8s.io/v1 [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:rules: {} [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - '' [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - pods [e2e-llm-inference-service] - verbs: [e2e-llm-inference-service] - get [e2e-llm-inference-service] - list [e2e-llm-inference-service] - watch [e2e-llm-inference-service] attributeRestrictions: null [e2e-llm-inference-service] apiGroups: [e2e-llm-inference-service] - inference.networking.x-k8s.io [e2e-llm-inference-service] - inference.networking.k8s.io [e2e-llm-inference-service] resources: [e2e-llm-inference-service] - inferencepools [e2e-llm-inference-service] apiVersion: authorization.openshift.io/v1 [e2e-llm-inference-service] kind: Role [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81801' [e2e-llm-inference-service] uid: 99fdb297-5179-492a-9e6a-8880578537e8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] annotations: [e2e-llm-inference-service] serving.kserve.io/inference-pool-migrated: v1 [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] generation: 2 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:annotations: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:serving.kserve.io/inference-pool-migrated: {} [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] - apiVersion: gateway.networking.k8s.io/v1beta1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81801' [e2e-llm-inference-service] uid: 99fdb297-5179-492a-9e6a-8880578537e8 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] parentRefs: [e2e-llm-inference-service] - group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] rules: [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/chat/completions [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/chat/completions [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/chat/completions/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/responses [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/responses [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/responses/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: /v1/messages [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test/v1/messages [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: inference.networking.k8s.io [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: /v1/messages/ [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] filters: [e2e-llm-inference-service] - type: URLRewrite [e2e-llm-inference-service] urlRewrite: [e2e-llm-inference-service] path: [e2e-llm-inference-service] replacePrefixMatch: / [e2e-llm-inference-service] type: ReplacePrefixMatch [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: /kserve-ci-e2e-test/scheduler-v06-pd-migration-test [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] - backendRefs: [e2e-llm-inference-service] - group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] port: 8000 [e2e-llm-inference-service] weight: 1 [e2e-llm-inference-service] matches: [e2e-llm-inference-service] - headers: [e2e-llm-inference-service] - name: X-Gateway-Model-Name [e2e-llm-inference-service] type: Exact [e2e-llm-inference-service] value: publishers/kserve-ci-e2e-test/models/facebook/opt-125m [e2e-llm-inference-service] path: [e2e-llm-inference-service] type: PathPrefix [e2e-llm-inference-service] value: / [e2e-llm-inference-service] timeouts: [e2e-llm-inference-service] backendRequest: 0s [e2e-llm-inference-service] request: 0s [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] message: Route was valid [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] message: All references resolved [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] controllerName: openshift.io/gateway-controller/v1 [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] message: Object affected by AuthPolicy [kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] openshift-ingress/openshift-ai-inference-authn] [e2e-llm-inference-service] observedGeneration: 2 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: kuadrant.io/AuthPolicyAffected [e2e-llm-inference-service] controllerName: kuadrant.io/policy-controller [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:appProtocol: {} [e2e-llm-inference-service] f:endpointPickerRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:port: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:number: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:matchLabels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPorts: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] - apiVersion: inference.networking.k8s.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:parents: {} [e2e-llm-inference-service] manager: pilot-discovery [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81788' [e2e-llm-inference-service] uid: 1400e18c-37cf-47c1-b43c-976ee17ec31f [e2e-llm-inference-service] spec: [e2e-llm-inference-service] appProtocol: http [e2e-llm-inference-service] endpointPickerRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] port: [e2e-llm-inference-service] number: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] matchLabels: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPorts: [e2e-llm-inference-service] - number: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parents: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] message: Referenced by an HTTPRoute accepted by the parentRef Gateway [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] message: Referenced ExtensionRef resolved successfully [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] reason: ResolvedRefs [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: ResolvedRefs [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: networking.istio.io [e2e-llm-inference-service] kind: Gateway [e2e-llm-inference-service] name: openshift-ai-inference [e2e-llm-inference-service] namespace: openshift-ingress [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] kind: AuthPolicy [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-policies [e2e-llm-inference-service] app.kubernetes.io/managed-by: odh-model-controller [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/managed-by: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:rules: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:authentication: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:public: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:anonymous: {} [e2e-llm-inference-service] f:credentials: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:overrides: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:fairness: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:value: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:response: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:success: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:headers: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:x-gateway-inference-fairness-id: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:x-gateway-inference-objective: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:metrics: {} [e2e-llm-inference-service] f:plain: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:expression: {} [e2e-llm-inference-service] f:priority: {} [e2e-llm-inference-service] f:targetRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] - apiVersion: kuadrant.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:status: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:conditions: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"type":"Accepted"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] k:{"type":"Enforced"}: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:lastTransitionTime: {} [e2e-llm-inference-service] f:message: {} [e2e-llm-inference-service] f:reason: {} [e2e-llm-inference-service] f:status: {} [e2e-llm-inference-service] f:type: {} [e2e-llm-inference-service] f:observedGeneration: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] subresource: status [e2e-llm-inference-service] time: '2026-07-01T22:03:29Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route-authn [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81663' [e2e-llm-inference-service] uid: 2c21b534-18ed-4fe7-95de-e6a66e88c7d1 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] rules: [e2e-llm-inference-service] authentication: [e2e-llm-inference-service] public: [e2e-llm-inference-service] anonymous: {} [e2e-llm-inference-service] credentials: {} [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] overrides: [e2e-llm-inference-service] fairness: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] objective: [e2e-llm-inference-service] value: unauthenticated [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] response: [e2e-llm-inference-service] success: [e2e-llm-inference-service] headers: [e2e-llm-inference-service] x-gateway-inference-fairness-id: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.fairness [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] x-gateway-inference-objective: [e2e-llm-inference-service] metrics: false [e2e-llm-inference-service] plain: [e2e-llm-inference-service] expression: auth.identity.objective [e2e-llm-inference-service] priority: 0 [e2e-llm-inference-service] targetRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: HTTPRoute [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-route [e2e-llm-inference-service] status: [e2e-llm-inference-service] conditions: [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:28Z' [e2e-llm-inference-service] message: AuthPolicy has been accepted [e2e-llm-inference-service] reason: Accepted [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] - lastTransitionTime: '2026-07-01T22:03:29Z' [e2e-llm-inference-service] message: AuthPolicy has been successfully enforced [e2e-llm-inference-service] reason: Enforced [e2e-llm-inference-service] status: 'True' [e2e-llm-inference-service] type: Enforced [e2e-llm-inference-service] observedGeneration: 1 [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81579' [e2e-llm-inference-service] uid: 185caa9a-f363-471d-9a96-5ddddeba1a0e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81792' [e2e-llm-inference-service] uid: ab0900f7-7a4a-466b-8fa5-f1bdf7bde4fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81596' [e2e-llm-inference-service] uid: ff9fa2b0-f454-4e93-bc62-95627ffcf802 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81579' [e2e-llm-inference-service] uid: 185caa9a-f363-471d-9a96-5ddddeba1a0e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81792' [e2e-llm-inference-service] uid: ab0900f7-7a4a-466b-8fa5-f1bdf7bde4fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1beta1 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81596' [e2e-llm-inference-service] uid: ff9fa2b0-f454-4e93-bc62-95627ffcf802 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-scheduler [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81579' [e2e-llm-inference-service] uid: 185caa9a-f363-471d-9a96-5ddddeba1a0e [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-epp-service.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-shadow-service [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:35Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-shadow-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81792' [e2e-llm-inference-service] uid: ab0900f7-7a4a-466b-8fa5-f1bdf7bde4fb [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-inference-pool-ip-2bbd07cf.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: networking.istio.io/v1alpha3 [e2e-llm-inference-service] kind: DestinationRule [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] llm-d.ai/managed: 'true' [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: networking.istio.io/v1 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:llm-d.ai/managed: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:exportTo: {} [e2e-llm-inference-service] f:host: {} [e2e-llm-inference-service] f:trafficPolicy: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:tls: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:insecureSkipVerify: {} [e2e-llm-inference-service] f:mode: {} [e2e-llm-inference-service] f:sni: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:27Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-workload-svc [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81596' [e2e-llm-inference-service] uid: ff9fa2b0-f454-4e93-bc62-95627ffcf802 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] exportTo: [e2e-llm-inference-service] - '*' [e2e-llm-inference-service] host: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] trafficPolicy: [e2e-llm-inference-service] tls: [e2e-llm-inference-service] insecureSkipVerify: true [e2e-llm-inference-service] mode: SIMPLE [e2e-llm-inference-service] sni: scheduler-v06-pd-migration-test-kserve-workload-svc.kserve-ci-e2e-test.svc.cluster.local [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] kind: InferencePool [e2e-llm-inference-service] metadata: [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] generation: 1 [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] managedFields: [e2e-llm-inference-service] - apiVersion: inference.networking.x-k8s.io/v1alpha2 [e2e-llm-inference-service] fieldsType: FieldsV1 [e2e-llm-inference-service] fieldsV1: [e2e-llm-inference-service] f:metadata: [e2e-llm-inference-service] f:labels: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/component: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:ownerReferences: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] k:{"uid":"479dcd87-3e82-4860-9050-eeae1af22ec3"}: {} [e2e-llm-inference-service] f:spec: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:extensionRef: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:failureMode: {} [e2e-llm-inference-service] f:group: {} [e2e-llm-inference-service] f:kind: {} [e2e-llm-inference-service] f:name: {} [e2e-llm-inference-service] f:portNumber: {} [e2e-llm-inference-service] f:selector: [e2e-llm-inference-service] .: {} [e2e-llm-inference-service] f:app.kubernetes.io/name: {} [e2e-llm-inference-service] f:app.kubernetes.io/part-of: {} [e2e-llm-inference-service] f:kserve.io/component: {} [e2e-llm-inference-service] f:targetPortNumber: {} [e2e-llm-inference-service] manager: manager [e2e-llm-inference-service] operation: Update [e2e-llm-inference-service] time: '2026-07-01T22:03:26Z' [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-inference-pool [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] ownerReferences: [e2e-llm-inference-service] - apiVersion: serving.kserve.io/v1alpha2 [e2e-llm-inference-service] blockOwnerDeletion: true [e2e-llm-inference-service] controller: true [e2e-llm-inference-service] kind: LLMInferenceService [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] uid: 479dcd87-3e82-4860-9050-eeae1af22ec3 [e2e-llm-inference-service] resourceVersion: '81528' [e2e-llm-inference-service] uid: 18d2089c-8502-4667-991d-13633261cad9 [e2e-llm-inference-service] spec: [e2e-llm-inference-service] extensionRef: [e2e-llm-inference-service] failureMode: FailOpen [e2e-llm-inference-service] group: '' [e2e-llm-inference-service] kind: Service [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-epp-service [e2e-llm-inference-service] portNumber: 9002 [e2e-llm-inference-service] selector: [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] targetPortNumber: 8000 [e2e-llm-inference-service] status: [e2e-llm-inference-service] parent: [e2e-llm-inference-service] - conditions: [e2e-llm-inference-service] - lastTransitionTime: '1970-01-01T00:00:00Z' [e2e-llm-inference-service] message: Waiting for controller [e2e-llm-inference-service] reason: Pending [e2e-llm-inference-service] status: Unknown [e2e-llm-inference-service] type: Accepted [e2e-llm-inference-service] parentRef: [e2e-llm-inference-service] group: gateway.networking.k8s.io [e2e-llm-inference-service] kind: Status [e2e-llm-inference-service] name: default [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:25:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: decode [e2e-llm-inference-service] pod-template-hash: 5c87756974 [e2e-llm-inference-service] timestamp: '2026-07-01T22:25:12Z' [e2e-llm-inference-service] window: 11.246s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: llm-d-routing-sidecar [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 9570069n [e2e-llm-inference-service] memory: 21096Ki [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16507968n [e2e-llm-inference-service] memory: 26304Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:25:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-workload-prefill [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] kserve.io/component: workload [e2e-llm-inference-service] llm-d.ai/role: prefill [e2e-llm-inference-service] pod-template-hash: 78b7f4c847 [e2e-llm-inference-service] timestamp: '2026-07-01T22:25:04Z' [e2e-llm-inference-service] window: 14.464s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 14421114n [e2e-llm-inference-service] memory: 24860Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1254 --- [e2e-llm-inference-service] INFO e2e.llmisvc.logging:test_llm_inference_service.py:1255 metadata: [e2e-llm-inference-service] name: scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k [e2e-llm-inference-service] namespace: kserve-ci-e2e-test [e2e-llm-inference-service] creationTimestamp: '2026-07-01T22:25:25Z' [e2e-llm-inference-service] labels: [e2e-llm-inference-service] app.kubernetes.io/component: llminferenceservice-router-scheduler [e2e-llm-inference-service] app.kubernetes.io/name: scheduler-v06-pd-migration-test [e2e-llm-inference-service] app.kubernetes.io/part-of: llminferenceservice [e2e-llm-inference-service] pod-template-hash: 59dc679cd [e2e-llm-inference-service] timestamp: '2026-07-01T22:25:12Z' [e2e-llm-inference-service] window: 16.891s [e2e-llm-inference-service] containers: [e2e-llm-inference-service] - name: main [e2e-llm-inference-service] usage: [e2e-llm-inference-service] cpu: 16846782n [e2e-llm-inference-service] memory: 30928Ki [e2e-llm-inference-service] apiVersion: metrics.k8s.io/v1beta1 [e2e-llm-inference-service] kind: PodMetrics [e2e-llm-inference-service] [e2e-llm-inference-service] ERROR e2e.llmisvc.logging:logging.py:48 [test_llm_inference_service] [2026-07-01T22:25:25.604440] end - ❌ 1323.163s: Service returned 502: [e2e-llm-inference-service] =============================== warnings summary =============================== [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] /workspace/source/python/kserve/.venv/lib64/python3.11/site-packages/pytest_asyncio/plugin.py:761: DeprecationWarning: The event_loop fixture provided by pytest-asyncio has been redefined in [e2e-llm-inference-service] /workspace/source/test/e2e/conftest.py:43 [e2e-llm-inference-service] Replacing the event_loop fixture with a custom implementation is deprecated [e2e-llm-inference-service] and will lead to errors in the future. [e2e-llm-inference-service] If you want to request an asyncio event loop with a scope other than function [e2e-llm-inference-service] scope, use the "scope" argument to the asyncio mark when marking the tests. [e2e-llm-inference-service] If you want to return different types of event loops, use the event_loop_policy [e2e-llm-inference-service] fixture. [e2e-llm-inference-service] [e2e-llm-inference-service] warnings.warn( [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-precise-prefix-cache-inline-config-workload-llmd-simulator-kvcache] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator2] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-gateway-ref-router-with-managed-route-model-fb-opt-125m-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf0] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m-with-lora-hf1] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-scheduler-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-pd-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-custom-route-timeout-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-pvc] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-with-refs-pd-scheduler-managed-workload-pd-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-no-scheduler-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py::test_llm_stop_feature[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service_stop.py:40: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_multi_node-router-managed-workload-simulated-dp-ep-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-inline-config-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-workload-llmd-simulator-model-qwen2.5-0.5b] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_tls.py::test_llm_tls_resources[cluster_cpu-cluster_single_node-router-managed-workload-single-cpu-model-fb-opt-125m] [e2e-llm-inference-service] llmisvc/test_llm_tls.py:93: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-configmap-ref-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-replicas-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-with-custom-template-workload-llmd-simulator] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-nonzero-threshold-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] llmisvc/test_llm_inference_service.py:245: PytestWarning: The test is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'. [e2e-llm-inference-service] @pytest.mark.llminferenceservice [e2e-llm-inference-service] [e2e-llm-inference-service] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [e2e-llm-inference-service] ---------- generated xml file: /workspace/artifacts-dir/junit_e2e.xml ---------- [e2e-llm-inference-service] --------------------------------- JSON report ---------------------------------- [e2e-llm-inference-service] report saved to: /workspace/artifacts-dir/e2e_results.json [e2e-llm-inference-service] =========================== short test summary info ============================ [e2e-llm-inference-service] FAILED llmisvc/test_llm_inference_service.py::test_llm_inference_service[cluster_cpu-cluster_single_node-router-managed-scheduler-v06-pd-config-migration-workload-llmd-simulator-pd] [e2e-llm-inference-service] ====== 1 failed, 41 passed, 3 skipped, 29 warnings in 5650.09s (1:34:10) ======= [must-gather] [must-gather ] OUT 2026-07-01T22:32:39.561416555Z Using must-gather plug-in image: quay.io/modh/must-gather:rhoai-2.24 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: dad42e20-1340-4ed9-80d3-35f48ac16f25 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-01T22:32:39.619951056Z namespace/openshift-must-gather-8xxfk created [must-gather] [must-gather ] OUT 2026-07-01T22:32:39.627635565Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-b2p86 created [must-gather] [must-gather ] OUT 2026-07-01T22:32:39.666121588Z pod for plug-in image quay.io/modh/must-gather:rhoai-2.24 created [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.434023308Z [disk usage checker] Started [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.437446782Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.697755104Z Error from server (NotFound): namespaces "redhat-ods-operator" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.866183446Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.866214168Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.866214168Z namespaces "redhat-ods-operator" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:46.869574207Z Error getting logs from redhat-ods-operator [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.036966783Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.037005147Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.037005147Z namespaces "redhat-ods-monitoring" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.040013847Z Error getting logs from redhat-ods-monitoring [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.203419011Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.203451521Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.203451521Z namespaces "redhat-ods-applications" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.206408561Z Error getting logs from redhat-ods-applications [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.368500253Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.368530288Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.368530288Z namespaces "rhods-notebooks" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.371499366Z Error getting logs from rhods-notebooks [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.539714521Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.539744922Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.539744922Z namespaces "rhoai-model-registries" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.543330032Z Error getting logs from rhoai-model-registries [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.720626523Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.720657660Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.720657660Z namespaces "istio-system" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.723558391Z Error getting logs from istio-system [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.882292250Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.882323147Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.882323147Z namespaces "knative-serving" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:47.885202459Z Error getting logs from knative-serving [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.042530031Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.042560110Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.042560110Z namespaces "redhat-ods-applications-auth-provider" not found [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.045683093Z Error getting logs from redhat-ods-applications-auth-provider [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.320627714Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.602759688Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.745311959Z error: the server doesn't have a resource type "auths" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.826934916Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.829461319Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:48.960871673Z error: the server doesn't have a resource type "monitorings" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.044382878Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.047179100Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.185522032Z error: the server doesn't have a resource type "featuretrackers" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.269884601Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.272612427Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.406297684Z error: the server doesn't have a resource type "codeflares" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.492303762Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.495257288Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.626617101Z error: the server doesn't have a resource type "dashboards" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.707524682Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.710277923Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.842462628Z error: the server doesn't have a resource type "datasciencepipelines" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.923833423Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:49.926681568Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.058807948Z error: the server doesn't have a resource type "feastoperators" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.140132027Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.143063536Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.277154857Z error: the server doesn't have a resource type "kserves" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.358797915Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.361516441Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.494365792Z error: the server doesn't have a resource type "kueues" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.578874217Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.581774864Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.713929560Z error: the server doesn't have a resource type "modelcontrollers" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.798380326Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.801195003Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:50.935218984Z error: the server doesn't have a resource type "modelmeshservings" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.015604654Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.018250383Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.154771483Z error: the server doesn't have a resource type "modelregistries" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.236893105Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.239747537Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.372257975Z error: the server doesn't have a resource type "rays" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.441918881Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.454519516Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.457145103Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.590891301Z error: the server doesn't have a resource type "trainingoperators" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.673823849Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.676503188Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.807693000Z error: the server doesn't have a resource type "trustyais" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.891529560Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:51.894281298Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.027340615Z error: the server doesn't have a resource type "workbenches" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.109460441Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.112345927Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.243919570Z error: the server doesn't have a resource type "hardwareprofiles" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.324030223Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.326722043Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.460336739Z error: the server doesn't have a resource type "llamastackoperators" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.541909849Z error: arguments in resource/name form must have a single resource and name [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:52.544759124Z Error collecting info from [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:53.272232849Z error: the server doesn't have a resource type "predictors" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:53.451346741Z error: the server doesn't have a resource type "localmodelnodegroups" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:53.580615622Z error: the server doesn't have a resource type "smcp" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:53.709023779Z error: the server doesn't have a resource type "smm" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:53.838249522Z error: the server doesn't have a resource type "smmr" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:54.422739035Z error: the server doesn't have a resource type "knativeservings" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:54.554846732Z error: the server doesn't have a resource type "configurations" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:54.686583892Z error: the server doesn't have a resource type "routes" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:54.818997289Z error: the server doesn't have a resource type "services" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:54.948209682Z error: the server doesn't have a resource type "revisions" [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:55.487795256Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:55.819427286Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:56.446514504Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:56.514659046Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:56.887540080Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:57.054722813Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:57.379674357Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:58.118545031Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:58.476177637Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:58.637407467Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:58.922533389Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:59.620988148Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:32:59.998706164Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:00.161546130Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:00.426000242Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:01.130327205Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:01.451044934Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:01.497376427Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:01.662813823Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:01.970095988Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:02.727207492Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:03.062828069Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:03.230709039Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:03.510557489Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:04.195568388Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:04.557950498Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:04.724835103Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:04.996203984Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:05.690084775Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:06.036933546Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:06.207998201Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:06.455712478Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:06.469935467Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:07.278444982Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:07.644466928Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:07.814468270Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:08.096458679Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:08.843903175Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:09.218536385Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:09.392330044Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:09.679262178Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:10.366611973Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:10.715628702Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:10.889275047Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:11.210490463Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:11.460129646Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:11.963583173Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:12.314615145Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:12.481079703Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:12.775410372Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:13.473507539Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:13.827185113Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:13.987131335Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:14.249325682Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:14.929527560Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:15.288637465Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:15.453380586Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:15.751445504Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:16.428445373Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:16.465684100Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:16.786369697Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:16.955216927Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:17.228998486Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:17.933114225Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:18.274988326Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:18.440146544Z Gathering data for ns/kserve-ci-e2e-test... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:18.710927249Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:19.402298560Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:19.778744800Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:19.942198147Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:20.901000225Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:21.050734254Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:21.416328824Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:21.471045714Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:21.584432640Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:22.569016182Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:22.735665918Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:23.109933358Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:23.271433110Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:24.265491885Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:24.414693968Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:24.792412719Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:24.954913775Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:25.916340849Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:26.070267395Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:26.434720594Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:26.476418179Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:26.601946864Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:27.139205349Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:27.330951040Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:27.719437243Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:27.882029321Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:28.454722340Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:28.653582895Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:29.053686447Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:29.222671896Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:29.781874011Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:29.957720372Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:30.347522832Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:30.511908999Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:31.043666863Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:31.222328865Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:31.480882965Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:31.610668124Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:31.769009489Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:32.371481151Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:32.550096981Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:32.935178214Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:33.100756510Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:33.645289599Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:33.819956222Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:34.207935629Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:34.370802939Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:34.942684427Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:35.126414195Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:35.517385496Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:35.683332538Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:36.251017312Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:36.479280887Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:36.486411704Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:36.872382541Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:37.036777421Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:37.566469028Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:37.752283956Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:38.146588887Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:38.306622023Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:38.847995968Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:39.024605246Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:39.423451525Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:39.585131040Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:40.120981693Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:40.321601383Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:40.715667230Z Wrote inspect data to must-gather. [must-gather] [must-gather-6xkr6] POD 2026-07-01T22:33:40.736358388Z Caches written to disk [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:41.746938421Z waiting for gather to complete [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:41.750740372Z downloading gather output [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.04244748Z receiving incremental file list [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.060151045Z ./ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.060197726Z aggregated-discovery-api.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.060522244Z aggregated-discovery-apis.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.061341863Z event-filter.html [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.063736821Z timestamp [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.063864934Z version [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064687013Z cluster-scoped-resources/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064706094Z cluster-scoped-resources/datasciencecluster.opendatahub.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064713044Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064747985Z cluster-scoped-resources/datasciencecluster.opendatahub.io/datascienceclusters/test-dsc.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064863318Z cluster-scoped-resources/dscinitialization.opendatahub.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064873198Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064914909Z cluster-scoped-resources/dscinitialization.opendatahub.io/dscinitializations/test-dsci.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064983871Z namespaces/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.064991181Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065065753Z namespaces/kserve-ci-e2e-test/kserve-ci-e2e-test.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065144704Z namespaces/kserve-ci-e2e-test/apps.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065193316Z namespaces/kserve-ci-e2e-test/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065242127Z namespaces/kserve-ci-e2e-test/apps/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065290458Z namespaces/kserve-ci-e2e-test/apps/daemonsets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065433282Z namespaces/kserve-ci-e2e-test/apps/deployments.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.065755979Z namespaces/kserve-ci-e2e-test/apps/replicasets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066059606Z namespaces/kserve-ci-e2e-test/apps/statefulsets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066115528Z namespaces/kserve-ci-e2e-test/autoscaling/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066153979Z namespaces/kserve-ci-e2e-test/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.06620575Z namespaces/kserve-ci-e2e-test/batch/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066247021Z namespaces/kserve-ci-e2e-test/batch/cronjobs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066329543Z namespaces/kserve-ci-e2e-test/batch/jobs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066366424Z namespaces/kserve-ci-e2e-test/build.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066395665Z namespaces/kserve-ci-e2e-test/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066483367Z namespaces/kserve-ci-e2e-test/build.openshift.io/builds.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066528988Z namespaces/kserve-ci-e2e-test/core/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066557449Z namespaces/kserve-ci-e2e-test/core/configmaps.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066719472Z namespaces/kserve-ci-e2e-test/core/endpoints.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.066814985Z namespaces/kserve-ci-e2e-test/core/events.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07162436Z namespaces/kserve-ci-e2e-test/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.071685301Z namespaces/kserve-ci-e2e-test/core/pods.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.071979388Z namespaces/kserve-ci-e2e-test/core/replicationcontrollers.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072127562Z namespaces/kserve-ci-e2e-test/core/secrets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072378738Z namespaces/kserve-ci-e2e-test/core/services.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07246287Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072509391Z namespaces/kserve-ci-e2e-test/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072590253Z namespaces/kserve-ci-e2e-test/image.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072628184Z namespaces/kserve-ci-e2e-test/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072661635Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072701056Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072784958Z namespaces/kserve-ci-e2e-test/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072817739Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072852529Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072921171Z namespaces/kserve-ci-e2e-test/networking.k8s.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072946712Z namespaces/kserve-ci-e2e-test/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072991063Z namespaces/kserve-ci-e2e-test/pods/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.072998913Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.073059594Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/router-gateway-1-openshift-default-54c47cdfc7-ss4x6.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.073171707Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/istio-proxy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.073178677Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/istio-proxy/istio-proxy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.073182657Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.073208508Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074271374Z namespaces/kserve-ci-e2e-test/pods/router-gateway-1-openshift-default-54c47cdfc7-ss4x6/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074353325Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074406727Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/router-gateway-2-openshift-default-684dc9b4b9-j47sj.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07451784Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/istio-proxy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07452877Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/istio-proxy/istio-proxy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07453476Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074574871Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074726874Z namespaces/kserve-ci-e2e-test/pods/router-gateway-2-openshift-default-684dc9b4b9-j47sj/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074825687Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074895138Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074987721Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/llm-d-routing-sidecar/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.074996171Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/llm-d-routing-sidecar/llm-d-routing-sidecar/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075000841Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075043932Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075286278Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075354329Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/llm-d-routing-sidecar/llm-d-routing-sidecar/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07537536Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/main/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07538263Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/main/main/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07538844Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/main/main/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075437111Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/main/main/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075543264Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/main/main/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075612066Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-5c87756974-j2gzz/main/main/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075650667Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075685397Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075769339Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/main/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07579455Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/main/main/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07580006Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/main/main/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075826871Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/main/main/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075917663Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/main/main/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.075998965Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-hhhsk/main/main/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076016265Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076087887Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076165189Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/main/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076172629Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/main/main/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076176639Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/main/main/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07620992Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/main/main/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076362034Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/main/main/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076429795Z namespaces/kserve-ci-e2e-test/pods/scheduler-v06-pd-migration-test-kserve-router-scheduler-59zfr9k/main/main/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076470896Z namespaces/kserve-ci-e2e-test/policy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076506607Z namespaces/kserve-ci-e2e-test/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076543068Z namespaces/kserve-ci-e2e-test/route.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076579009Z namespaces/kserve-ci-e2e-test/route.openshift.io/routes.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076723952Z namespaces/kuadrant-system/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076771423Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076830255Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076877196Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076935438Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.076982748Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077109732Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077488601Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077802788Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077846169Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077904291Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077934782Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.077988593Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078093025Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078121466Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078177897Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078250179Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078288Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078324711Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078477175Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.078590337Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.079004677Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.07910751Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.079451528Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.079575491Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.08037925Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080518713Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080553874Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080693028Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080743239Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.08077896Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080824161Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080920393Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080952154Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.080984195Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081056176Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081088127Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081149859Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081160089Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.0811945Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino-7c77957575-nf65n.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081258961Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081269051Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081273522Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.081321033Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.085994395Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086074807Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086097927Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086150739Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/authorino-operator-6d85f6564-s4vc7.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.08623321Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086250541Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086254961Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086264241Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086469076Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086533378Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086553008Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086602619Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/dns-operator-controller-manager-65b49595d7-c59jh.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086693532Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086709282Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086714142Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086722322Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086833585Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086908097Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086938927Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.086972528Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin-85fd4b8b9d-pgv8h.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087073641Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087086951Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087090661Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087116722Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087195044Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087261835Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087301326Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087352017Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.08744986Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.08745838Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.08746855Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.087500251Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.10788107Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.107914691Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.107977072Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108002263Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador-limitador-69574b596d-8mmn4.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108119886Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108136126Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108141946Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108170627Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108251939Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.10832168Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108355531Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108401722Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/limitador-operator-controller-manager-6f9f468797-f448h.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108489615Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108499365Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108503535Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108545986Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.10871989Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108784772Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108799792Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108852243Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108901674Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108942555Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.108975076Z namespaces/openshift-ingress/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109063328Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.10913069Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109169721Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109222242Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109258923Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109337165Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.10956093Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109755495Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109800596Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109847097Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.109901008Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.10995189Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110023211Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110088353Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110122394Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110207716Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110241746Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110289718Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.110703948Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.11081602Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111063566Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111147948Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111389974Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111511557Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111695651Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111778603Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111813144Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111903716Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111941087Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.111972658Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112021299Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112130342Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112170253Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112201124Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112254375Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112291426Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112338807Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112346387Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112393798Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/istiod-openshift-gateway-94bb8fbfd-kqnw7.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112491571Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112506081Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112510571Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.112530112Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.170996484Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171083236Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171099197Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171151818Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/openshift-ai-inference-openshift-default-655f7c58b4-57g8x.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171260601Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171275161Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171280521Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.171298871Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.176705131Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.177889289Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.17791984Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router-default-bcd57767c-rwpfk.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178052153Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178062684Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178066534Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178103174Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/current.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178225288Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/previous.insecure.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178295009Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/previous.log [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178310579Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178364751Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178402782Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.178459553Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.197582932Z [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.197604402Z sent 3,347 bytes received 2,062,853 bytes 1,377,466.67 bytes/sec [must-gather] [must-gather-6xkr6] OUT 2026-07-01T22:33:42.197609893Z total size is 24,854,589 speedup is 12.03 [must-gather] [must-gather ] OUT 2026-07-01T22:33:42.433752428Z namespace/openshift-must-gather-8xxfk deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: dad42e20-1340-4ed9-80d3-35f48ac16f25 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-01T22:33:42.562681371Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e589877e8cc36f2773ea2a7f13a0090e4ec912c558ed652b58e44baf7cef7ad6 [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: dad42e20-1340-4ed9-80d3-35f48ac16f25 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [must-gather] [must-gather ] OUT 2026-07-01T22:33:42.584945035Z namespace/openshift-must-gather-z7v9q created [must-gather] [must-gather ] OUT 2026-07-01T22:33:42.593448219Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-ck7c2 created [must-gather] [must-gather ] OUT 2026-07-01T22:33:42.618877289Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e589877e8cc36f2773ea2a7f13a0090e4ec912c558ed652b58e44baf7cef7ad6 created [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:43.886345025Z [disk usage checker] Started [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:43.890247840Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:44.109775655Z Gathering data for ns/openshift-cluster-version... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:44.263404436Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:44.366185971Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:44.463521049Z Gathering data for ns/default... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:44.820274406Z Gathering data for ns/openshift... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:45.289316680Z Gathering data for ns/kube-system... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.082591918Z Gathering data for ns/openshift-etcd... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.473772269Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.473807260Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.473807260Z namespaces "assisted-installer" not found [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.818291340Z Waiting on subprocesses to finish execution. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.825501605Z INFO: Gathering machine config daemon's old logs from all nodes [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.828690853Z WARNING: Collecting one or more kube-apiserver related logs on ALL masters in your cluster. This could take a large amount of time. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.830061698Z INFO: Gathering on-disk MachineConfig from degraded nodes [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.837075155Z Executing Istio gather script [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.840063630Z INFO: Gathering HAProxy config files [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.862063553Z INFO: Collecting host service logs for crio [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.862387255Z INFO: Collecting host service logs for kubelet [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.862682203Z INFO: Collecting host service logs for rpm-ostreed [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.862982402Z INFO: Collecting host service logs for ostree-finalize-staged [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.863281419Z INFO: Collecting host service logs for machine-config-daemon-firstboot [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.863556286Z INFO: Collecting host service logs for machine-config-daemon-host [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.864027366Z INFO: Collecting host service logs for NetworkManager [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.864316750Z INFO: Collecting host service logs for openvswitch [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.864653494Z INFO: Collecting host service logs for ovs-configuration [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.865318365Z INFO: Collecting host service logs for ovsdb-server [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.865769400Z INFO: Collecting host service logs for ovs-vswitchd [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.866297776Z INFO: Waiting for worker host service log collection to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:46.935910924Z INFO: Waiting for node performance related collection to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.383849443Z INFO: Collecting Insights Archives from insights-operator-5f67b55555-zs75b [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.383906658Z insights-runtime-extractor-45r8m [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.383927297Z insights-runtime-extractor-sk28n [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.383947353Z insights-runtime-extractor-wpqxs [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.523298383Z No resources found in openshift-etcd namespace. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.626777449Z INFO: Found 1 replicas - prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.723562687Z INFO: Waiting for on-disk MachineConfig collection to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.723591022Z INFO: on-disk MachineConfig config collection complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.744298740Z error: the server doesn't have a resource type "performanceprofile" [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.939771801Z error: only SOURCE_DIR and POD:DESTINATION_DIR should be specified as arguments [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:47.939771801Z See 'oc rsync -h' for help and examples [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.095279854Z INFO: "metallb-operator" not detected. Skipping. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.194087666Z INFO: namespace openshift-frr-k8s not detected. Skipping. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.256393813Z INFO: "sriov-network-operator" not detected. Skipping. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.281039834Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.297761749Z INFO: "kubernetes-nmstate-operator" not detected. Skipping. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.322415635Z No resources found [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.412630032Z INFO: Worker host service log collection to complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.419198309Z INFO: Getting alertmanagers from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.473111274Z INFO: Waiting for HAProxy config collection to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.784814220Z ERROR: No running kube-apiserver pods found [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:48.980354863Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.146842264Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.377923725Z error: the server doesn't have a resource type "multi-networkpolicy" [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.444664236Z INFO: 'previous-logs' folder not found on ip-10-0-132-117.ec2.internal, skipping... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.555026746Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.604673232Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.611687220Z INFO: HAProxy config collection complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.695400525Z No resources found [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.701957421Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.732261498Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:49.933247176Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.006677323Z error: the server doesn't have a resource type "machineconfigs" [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.139900838Z INFO: Getting rules from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.192322430Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.269317865Z Gathering data for ns/kuadrant-system... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.380585190Z error: the server doesn't have a resource type "machineconfigpools" [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.387524361Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.436035685Z INFO: 'previous-logs' folder not found on ip-10-0-138-219.ec2.internal, skipping... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.448405435Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.805201018Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.808391450Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.845742869Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.943895219Z INFO: Getting status/config from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.947523283Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.993595841Z INFO: 'previous-logs' folder not found on ip-10-0-140-104.ec2.internal, skipping... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.993707480Z INFO: Waiting for Machine Config Daemon termination log collection to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:50.993760413Z INFO: Machine Config Daemon termination log collection complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.220419211Z error: the server doesn't have a resource type "kubeletconfigs" [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.259192136Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.269438781Z Inspecting resource clusterrole.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.301249828Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.413137584Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.605306508Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.663686451Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.667542435Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istio-reader-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.677125093Z INFO: OLM v1 CRDs not detected. Skipping OLM v1 resource collection. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.745561436Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.758801656Z INFO: Getting status/flags from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.995727335Z INFO: INTERCONNECT MODE [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:51.995832129Z INFO: Gathering ovn-kubernetes DBs [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.051883050Z Gathering data for ns/openshift-monitoring... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.052752669Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.055300881Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-clusterrole-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.074123140Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.141888356Z INFO: Getting status/runtimeinfo from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.207517775Z INFO: Gathering OVN_Northbound from ovnkube-node-lrvqz... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.207839912Z INFO: Gathering OVN_Northbound from ovnkube-node-q2zrq... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.208134694Z INFO: Gathering OVN_Northbound from ovnkube-node-sw9rx... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.208506552Z INFO: Gathering OVN_Southbound from ovnkube-node-lrvqz... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.208799786Z INFO: Gathering OVN_Southbound from ovnkube-node-q2zrq... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.209103487Z INFO: Gathering OVN_Southbound from ovnkube-node-sw9rx... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.483651537Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.489450019Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.496561643Z Inspecting resource clusterrolebinding.rbac.authorization.k8s.io/istiod-gateway-controller-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.579771054Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.590986008Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.601337231Z INFO: Getting targets?state=active from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.602279503Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.627569307Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.665946695Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.699299112Z tar: Removing leading `/' from member names [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.834552591Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.888027887Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:52.920067615Z INFO: Getting status/tsdb from prometheus-k8s-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.210264361Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.261331223Z INFO: Waiting for network log collection to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.261382963Z INFO: Waiting for ovnk database copies to complete ... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.262405614Z INFO: Copying ovnk databases complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.264584169Z 7.2M must-gather/network_logs/ovnk_database_store [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.267542657Z ovnk_database_store/ [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.267542657Z ovnk_database_store/ovnkube-node-sw9rx_sbdb [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.321308227Z ovnk_database_store/ovnkube-node-lrvqz_nbdb [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.339311816Z ovnk_database_store/ovnkube-node-q2zrq_nbdb [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.356224655Z ovnk_database_store/ovnkube-node-sw9rx_nbdb [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.372025049Z ovnk_database_store/ovnkube-node-lrvqz_sbdb [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.400696385Z INFO: Getting status from alertmanager-main-0 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.423612398Z ovnk_database_store/ovnkube-node-q2zrq_sbdb [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.484315850Z INFO: Network log collection complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.564591221Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:53.847595683Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.002629302Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.147206435Z Error from server (NotFound): deployments.apps "cluster-node-tuning-operator" not found [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.149971157Z INFO: Fallback to identify the container image from release info [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.231696257Z W0701 22:33:54.231647 329 util.go:195] skipping , failed to read event err: Object 'Kind' is missing in '' [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.240060807Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.505130345Z Gathering data for ns/openshift-network-console... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.675537507Z INFO: Image with low level tools to use: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2c4f9949a07a5338dd10a5abcb8d6d3f5eebe93333139869f9eec8802800162f [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.804605966Z daemonset.apps/perf-node-gather-daemonset created [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:54.956928133Z Waiting for performance profile collector pods to become ready: 1 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.032083828Z Gathering data for ns/openshift-console-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.063293156Z Inspecting resource crd/authorizationpolicies.security.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.284215788Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.512322928Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.517315018Z Inspecting resource crd/destinationrules.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.574912088Z Gathering data for ns/openshift-console... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:55.774207898Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.008583898Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.014147179Z Inspecting resource crd/envoyfilters.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.128059941Z Waiting for performance profile collector pods to become ready: 2 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.191820958Z Gathering data for ns/openshift-cluster-storage-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.223181142Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.444546465Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.447658823Z Inspecting resource crd/gatewayclasses.gateway.networking.k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.647406134Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.779738806Z Gathering data for ns/openshift-dns-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.890154609Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:56.892671272Z Inspecting resource crd/gateways.gateway.networking.k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.097996853Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.156067356Z Gathering data for ns/openshift-dns... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.268250544Z Waiting for performance profile collector pods to become ready: 3 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.322473771Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.324908682Z Inspecting resource crd/gateways.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.517721562Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.732636449Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.736163833Z Inspecting resource crd/grpcroutes.gateway.networking.k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.829669738Z Gathering data for ns/openshift-image-registry... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:57.943061128Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.160599200Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.162922555Z Inspecting resource crd/httproutes.gateway.networking.k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.405881528Z Gathering data for ns/openshift-ingress-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.424447316Z Waiting for performance profile collector pods to become ready: 4 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.430188646Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.664333229Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.667356835Z Inspecting resource crd/inferencemodelrewrites.inference.networking.x-k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.813090441Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:58.861468171Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.009662270Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.085121751Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.088935891Z Inspecting resource crd/inferenceobjectives.inference.networking.x-k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.277380075Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.513835670Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.516782709Z Inspecting resource crd/inferencepoolimports.inference.networking.x-k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.592705848Z Waiting for performance profile collector pods to become ready: 5 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.725687758Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.939871285Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:33:59.945111630Z Inspecting resource crd/inferencepools.inference.networking.k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.141683314Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.185973507Z Gathering data for ns/openshift-ingress-canary... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.375094370Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.377531792Z Inspecting resource crd/inferencepools.inference.networking.x-k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.563294408Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.670712665Z Gathering data for ns/openshift-insights... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.741874178Z Waiting for performance profile collector pods to become ready: 6 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.794919072Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.797688440Z Inspecting resource crd/peerauthentications.security.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:00.965361718Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.187308030Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.190697298Z Inspecting resource crd/proxyconfigs.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.377512485Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.581001973Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.583495815Z Inspecting resource crd/referencegrants.gateway.networking.k8s.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.743610801Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.885876000Z Waiting for performance profile collector pods to become ready: 7 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.963954104Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:01.966236491Z Inspecting resource crd/requestauthentications.security.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.144481756Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.356073989Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.358649197Z Inspecting resource crd/serviceentries.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.543253375Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.756133969Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.759191061Z Inspecting resource crd/sidecars.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:02.958950018Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.026440809Z Daemonset perf-node-gather-daemonset ready 3 out of 3 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.204143384Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.206683610Z Inspecting resource crd/telemetries.telemetry.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.321356834Z Collecting performance related data for node ip-10-0-140-104.ec2.internal [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.328092405Z Collecting performance related data for node ip-10-0-132-117.ec2.internal [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.336241594Z Collecting performance related data for node ip-10-0-138-219.ec2.internal [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.435839349Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.685311210Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.689022509Z Inspecting resource crd/virtualservices.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.804402322Z Gathering data for ns/openshift-lws-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:03.953121650Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.021813953Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.237315400Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.240873235Z Inspecting resource crd/wasmplugins.extensions.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.525075549Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.601701406Z Gathering data for ns/kserve... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.807361452Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:04.811364631Z Inspecting resource crd/workloadentries.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:05.068819775Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:05.302849235Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:05.305028990Z Inspecting resource crd/workloadgroups.networking.istio.io [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:05.502006985Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:05.735167766Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:05.869680260Z Inspecting resource mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:06.137456837Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.087170574Z Gathering data for ns/openshift-config... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.197332763Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.370411503Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.520247347Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.616878451Z Gathering data for ns/openshift-config-managed... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.667394086Z Inspecting resource validatingwebhookconfiguration.admissionregistration.k8s.io/istio-validator-openshift-gateway-openshift-ingress [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:07.900009397Z Gathering data for ns/openshift-ingress... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:08.227269192Z Gathering data for ns/openshift-kube-apiserver-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:08.640885142Z Gathering data for ns/openshift-kube-apiserver... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:08.958628230Z Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.033667907Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.074481554Z Gathering data for ns/openshift-kube-controller-manager... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.179977103Z Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.329082423Z Wrote inspect data to /must-gather/istio. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.487970100Z Gathering data for ns/openshift-kube-controller-manager-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.862515049Z Done executing Istio gather script [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:09.877738264Z Gathering data for ns/openshift-kube-scheduler... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:10.196869523Z Gathering data for ns/openshift-kube-scheduler-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:10.581215749Z Gathering data for ns/openshift-kube-storage-version-migrator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:10.976232342Z Gathering data for ns/openshift-kube-storage-version-migrator-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:11.338384951Z Gathering data for ns/openshift-user-workload-monitoring... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:11.894647515Z Gathering data for ns/openshift-multus... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:12.036005650Z Collecting kubelet logs for node ip-10-0-140-104.ec2.internal [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:12.037784226Z Collecting kubelet logs for node ip-10-0-132-117.ec2.internal [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:12.039549929Z Collecting kubelet logs for node ip-10-0-138-219.ec2.internal [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:12.460248344Z daemonset.apps "perf-node-gather-daemonset" deleted from openshift-must-gather-z7v9q namespace [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:12.467136965Z INFO: Node performance data collection complete. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:13.565140140Z Gathering data for ns/openshift-ovn-kubernetes... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:14.038401151Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:14.955514466Z Gathering data for ns/openshift-host-network... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:15.283055525Z Gathering data for ns/openshift-network-diagnostics... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:15.864427320Z Gathering data for ns/openshift-network-node-identity... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:16.340581036Z Gathering data for ns/openshift-network-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:16.836006302Z Gathering data for ns/openshift-cloud-network-config-controller... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:17.164731689Z Gathering data for ns/openshift-cluster-node-tuning-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:17.685897015Z Gathering data for ns/openshift-apiserver-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:17.992458028Z Gathering data for ns/openshift-apiserver... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:18.460524149Z Gathering data for ns/openshift-controller-manager-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:18.747151241Z Gathering data for ns/openshift-controller-manager... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:19.042690125Z [disk usage checker] Volume usage percentage: current = 18 ; allowed = 70 [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:19.048409939Z Gathering data for ns/openshift-cluster-samples-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:19.775288722Z Gathering data for ns/openshift-operator-lifecycle-manager... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:20.121106034Z Gathering data for ns/openshift-service-ca-operator... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:20.470715293Z Gathering data for ns/openshift-service-ca... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:20.885540221Z Gathering data for ns/openshift-cluster-csi-drivers... [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:22.027551470Z Wrote inspect data to must-gather. [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:22.027597833Z error: inspection completed with the errors occurred while gathering data: [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:22.027597833Z skipping gathering secrets/support due to error: secrets "support" not found [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:22.160552636Z error: the server doesn't have a resource type "clusters" [must-gather] [must-gather-h9gxr] POD 2026-07-01T22:34:22.345095122Z Caches written to disk [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.357911526Z waiting for gather to complete [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.361242827Z downloading gather output [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.644592892Z receiving incremental file list [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.668338058Z ./ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.668488482Z aggregated-discovery-api.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.668879872Z aggregated-discovery-apis.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.669798014Z event-filter.html [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.672058929Z timestamp [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.672091409Z version [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.67913541Z cluster-scoped-resources/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679162911Z cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679172091Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679221852Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/default-network-annotation.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679339175Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/servicecidrs.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679420667Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicies/user-defined-networks-namespace-label.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679491589Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.6795381Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/default-network-annotation-binding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679621102Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/servicecidrs-binding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679700894Z cluster-scoped-resources/admissionregistration.k8s.io/validatingadmissionpolicybindings/user-defined-networks-namespace-label-binding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679798677Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679852208Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/multus.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.679960561Z cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/network-node-identity.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.680040272Z cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.680397401Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.680445052Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/accounts.nim.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.680613606Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.680823211Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/adminpolicybasedexternalroutes.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.681021866Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertingrules.monitoring.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.68116807Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagerconfigs.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.684389778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertmanagers.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.685871444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/alertrelabelconfigs.monitoring.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.685980527Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyapprovals.devportal.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.68612214Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeyrequests.devportal.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.686234903Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apikeys.devportal.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.686385736Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiproducts.devportal.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.686912449Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apirequestcounts.apiserver.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.687083643Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/apiservers.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.687226607Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/appliedmanifestworks.work.open-cluster-management.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.687308309Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authconfigs.authorino.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.688086627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authentications.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.688313823Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorinos.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.688463637Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.688641551Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authpolicies.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.689787879Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/baselineadminnetworkpolicies.policy.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.689999744Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/builds.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.690185158Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/bundles.trust.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.690345732Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/catalogsources.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.69065573Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificaterequests.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.690776813Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certificates.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.690986008Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/certmanagers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.691267555Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/challenges.acme.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.69190599Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudcredentials.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692060754Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692167217Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/cloudprivateipconfigs.cloud.network.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692276999Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterclaims.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692371202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercloudeventsources.eventing.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692484484Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustercsidrivers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692633668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterimagepolicies.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.692812162Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterissuers.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.693591191Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteroperators.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.693720664Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterresourcequotas.quota.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.693845517Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterserviceversions.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.695739253Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterstoragecontainers.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.695897887Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clustertriggerauthentications.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.696080481Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusteruserdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.696306027Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/clusterversions.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.696571144Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.696988784Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.697132177Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/configs.samples.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.69724881Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleclidownloads.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.697355052Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleexternalloglinks.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.697461615Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolelinks.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.697576548Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolenotifications.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.69767579Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleplugins.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.697822024Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolequickstarts.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.697961627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.69809364Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoles.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.698361477Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consolesamples.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.69850456Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/consoleyamlsamples.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.698611603Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/controllerconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.699172047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/credentialsrequests.cloudcredential.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.699288809Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/csisnapshotcontrollers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.699405742Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/datascienceclusters.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.699611397Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.700375146Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.700484608Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnses.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.700670163Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnshealthcheckprobes.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.700777926Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnspolicies.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.700937209Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.ingress.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701068563Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dnsrecords.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701235627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/dscinitializations.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701350199Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressfirewalls.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701471392Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressips.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701609846Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressqoses.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701754449Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressrouters.network.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701878332Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/egressservices.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.701991665Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.702159179Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/etcds.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.702299172Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/featuregates.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.702425526Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.702569099Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.702935978Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.703133873Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.703677266Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/helmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.703791419Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.704569087Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httpscaledobjects.http.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.704698771Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentpolicies.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.704830694Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagecontentsourcepolicies.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.704936316Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagedigestmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.70509028Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepolicies.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.705587892Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagepruners.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.70591222Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/images.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.706126485Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/imagetagmirrorsets.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.706279049Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencegraphs.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.706455063Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.706594837Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.706711559Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.706865763Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.707038218Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.707176431Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.709588319Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/infrastructures.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.710117712Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresscontrollers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.710800829Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ingresses.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.711001294Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/insightsoperators.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.711189688Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/installplans.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.711317051Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/interceptorroutes.http.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.711450265Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ipamclaims.k8s.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.711557097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ippools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.71168627Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/issuers.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.712574472Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/istiocsrs.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.712881529Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kedacontrollers.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.716328783Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kuadrants.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.716462536Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeapiservers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.71661701Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubecontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.720360171Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubeschedulers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.720506924Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/kubestorageversionmigrators.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.720642428Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersetoperators.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.720794821Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/leaderworkersets.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.724166303Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/limitadors.limitador.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7244639Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceserviceconfigs.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.730535577Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/llminferenceservices.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.734602016Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfignodes.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.73474485Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineconfigurations.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735046487Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosbuilds.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735214711Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/machineosconfigs.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735362755Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735461677Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735610201Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/networks.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735857687Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodes.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.735968619Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/nodeslicepools.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.736102643Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oauths.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.736289467Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/oidcpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.73642195Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/olmconfigs.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.736528633Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftapiservers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.736643696Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/openshiftcontrollermanagers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.736758668Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorconditions.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.736903832Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorgroups.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737067096Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorhubs.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.73721662Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operatorpkis.network.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737321632Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/operators.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737453105Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/orders.acme.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737602829Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/overlappingrangeipreservations.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737706791Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737831305Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/pinnedimagesets.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.737938087Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/planpolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.738090651Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podmonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.738360747Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/podnetworkconnectivitychecks.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.73848433Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/probes.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.738728926Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/profiles.tuned.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.738842589Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projecthelmchartrepositories.helm.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.738977173Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/projects.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.739099395Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheuses.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.740981711Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/prometheusrules.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741127545Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxies.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741237647Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7413374Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rangeallocations.security.internal.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741437822Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741583536Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741696238Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741846202Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/rolebindingrestrictions.authorization.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.741962175Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledjobs.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.743499982Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/scaledobjects.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.743646136Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/schedulers.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.743762778Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/securitycontextconstraints.security.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.743902882Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicecas.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.744016625Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.744221049Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servicemonitors.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.744459355Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/servingruntimes.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.744979108Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.745274795Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storages.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.745391258Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storagestates.migration.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.745501731Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/storageversionmigrations.migration.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.745612343Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/subscriptions.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.746263039Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.746438843Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetrypolicies.extensions.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.746543836Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/thanosrulers.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.74796422Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tlspolicies.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.748119914Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tokenratelimitpolicies.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.748273018Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trainedmodels.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.74837699Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/triggerauthentications.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.748524444Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/trustmanagers.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.74879006Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/tuneds.tuned.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.748909003Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/userdefinednetworks.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.749090728Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.749477047Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumepopulators.populator.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.74958125Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotclasses.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.749696902Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshotcontents.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.749882097Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/volumesnapshots.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750040631Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750183214Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750318928Z cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750479152Z cluster-scoped-resources/apiregistration.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750701277Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750757338Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1..yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750862861Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.acme.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.750949783Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751041785Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751132177Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apiserver.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751278831Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751377893Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.apps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751462905Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751551998Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751635669Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.authorization.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751731062Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.autoscaling.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751807154Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.batch.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751890466Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.build.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.751983468Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752125052Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.certificates.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752208034Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloud.network.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752306856Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.cloudcredential.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752387538Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.config.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7524785Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.console.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752563872Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752655364Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.datasciencecluster.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752734516Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.discovery.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752821558Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.dscinitialization.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75290326Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.events.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.752992022Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.flowcontrol.apiserver.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753097345Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753184287Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.image.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753273049Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.imageregistry.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753361602Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.inference.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753452284Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.ingress.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753540806Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753627598Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.k8s.ovn.org.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75371865Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753806132Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.leaderworkerset.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753892194Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.machineconfiguration.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.753981006Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754080499Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.monitoring.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754164501Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.network.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754251873Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754334605Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754424797Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.nim.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754507949Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.node.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754594611Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.oauth.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754690004Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754769905Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.754872758Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.packages.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7549575Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.policy.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755094684Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.project.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755181286Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.quota.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755268588Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75535797Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.resource.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755443512Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.route.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755540184Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.samples.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755628577Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.scheduling.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755717348Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.internal.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755809141Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755900643Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.security.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.755993305Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.snapshot.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756110128Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75620288Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.telemetry.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756299293Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.template.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756386035Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.tuned.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756472327Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.user.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75658062Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1.work.open-cluster-management.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756682482Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.cluster.open-cluster-management.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756780794Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.controlplane.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756870087Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.devportal.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.756958089Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.eventing.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757065801Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757150803Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.extensions.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757248346Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.http.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757329468Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75742178Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.k8s.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757505742Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757598844Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757686206Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.limitador.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757782489Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.migration.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757873021Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.757963403Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operator.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758076966Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758171578Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.policy.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75825731Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758347442Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.telemetry.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758431914Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.trust.cert-manager.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758528867Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha1.whereabouts.cni.cncf.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758612539Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758702781Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758787083Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha2.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758886296Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1alpha3.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.758973467Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.admissionregistration.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759110691Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.external.metrics.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759213253Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759301906Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.helm.openshift.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759391498Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.http.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.75948686Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759569552Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759665724Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.monitoring.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759760627Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759844019Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.operator.authorino.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.759937991Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.populator.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760044934Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760160436Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.serving.kserve.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760250158Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76033412Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v1beta3.authorino.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760420193Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.autoscaling.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760519985Z cluster-scoped-resources/apiregistration.k8s.io/apiservices/v2.operators.coreos.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760573736Z cluster-scoped-resources/certificates.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760581457Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760610387Z cluster-scoped-resources/certificates.k8s.io/certificatesigningrequests/csr-ldnbf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760756511Z cluster-scoped-resources/config.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760805712Z cluster-scoped-resources/config.openshift.io/apiservers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.760967866Z cluster-scoped-resources/config.openshift.io/authentications.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76112483Z cluster-scoped-resources/config.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.761227382Z cluster-scoped-resources/config.openshift.io/clusterimagepolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.761295804Z cluster-scoped-resources/config.openshift.io/clusteroperators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.761607131Z cluster-scoped-resources/config.openshift.io/clusterversions.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.761715354Z cluster-scoped-resources/config.openshift.io/consoles.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.761820187Z cluster-scoped-resources/config.openshift.io/dnses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.761918619Z cluster-scoped-resources/config.openshift.io/featuregates.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762053672Z cluster-scoped-resources/config.openshift.io/imagecontentpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762146944Z cluster-scoped-resources/config.openshift.io/imagedigestmirrorsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762238867Z cluster-scoped-resources/config.openshift.io/imagepolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762322449Z cluster-scoped-resources/config.openshift.io/images.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762445232Z cluster-scoped-resources/config.openshift.io/imagetagmirrorsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762537074Z cluster-scoped-resources/config.openshift.io/infrastructures.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762638956Z cluster-scoped-resources/config.openshift.io/ingresses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762723749Z cluster-scoped-resources/config.openshift.io/networks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762821931Z cluster-scoped-resources/config.openshift.io/nodes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.762929994Z cluster-scoped-resources/config.openshift.io/oauths.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763015525Z cluster-scoped-resources/config.openshift.io/operatorhubs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763130658Z cluster-scoped-resources/config.openshift.io/projects.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763222751Z cluster-scoped-resources/config.openshift.io/proxies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763310783Z cluster-scoped-resources/config.openshift.io/schedulers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763399515Z cluster-scoped-resources/config.openshift.io/clusteroperators/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763442646Z cluster-scoped-resources/config.openshift.io/clusteroperators/console.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763539838Z cluster-scoped-resources/config.openshift.io/clusteroperators/csi-snapshot-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763635381Z cluster-scoped-resources/config.openshift.io/clusteroperators/dns.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763722243Z cluster-scoped-resources/config.openshift.io/clusteroperators/image-registry.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763807325Z cluster-scoped-resources/config.openshift.io/clusteroperators/ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.763906737Z cluster-scoped-resources/config.openshift.io/clusteroperators/insights.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76399692Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764188294Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764288467Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-scheduler.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764384319Z cluster-scoped-resources/config.openshift.io/clusteroperators/kube-storage-version-migrator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764472951Z cluster-scoped-resources/config.openshift.io/clusteroperators/monitoring.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764612324Z cluster-scoped-resources/config.openshift.io/clusteroperators/network.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764747608Z cluster-scoped-resources/config.openshift.io/clusteroperators/node-tuning.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76483551Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.764927762Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765020864Z cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765171978Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-catalog.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76526239Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager-packageserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765347332Z cluster-scoped-resources/config.openshift.io/clusteroperators/operator-lifecycle-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765432474Z cluster-scoped-resources/config.openshift.io/clusteroperators/service-ca.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765522137Z cluster-scoped-resources/config.openshift.io/clusteroperators/storage.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765589828Z cluster-scoped-resources/config.openshift.io/clusterversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765640029Z cluster-scoped-resources/config.openshift.io/clusterversions/version.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765707061Z cluster-scoped-resources/config.openshift.io/consoles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765747782Z cluster-scoped-resources/config.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765810904Z cluster-scoped-resources/config.openshift.io/featuregates/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765849474Z cluster-scoped-resources/config.openshift.io/featuregates/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765911776Z cluster-scoped-resources/config.openshift.io/infrastructures/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.765950637Z cluster-scoped-resources/config.openshift.io/infrastructures/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766002478Z cluster-scoped-resources/config.openshift.io/oauths/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76606657Z cluster-scoped-resources/config.openshift.io/oauths/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766122491Z cluster-scoped-resources/config.openshift.io/proxies/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766169372Z cluster-scoped-resources/config.openshift.io/proxies/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766221314Z cluster-scoped-resources/console.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766233664Z cluster-scoped-resources/console.openshift.io/consoleplugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766282285Z cluster-scoped-resources/console.openshift.io/consoleplugins/kuadrant-console-plugin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766367517Z cluster-scoped-resources/console.openshift.io/consoleplugins/monitoring-plugin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766445719Z cluster-scoped-resources/console.openshift.io/consoleplugins/networking-console-plugin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7664933Z cluster-scoped-resources/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76650147Z cluster-scoped-resources/core/nodes/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766557211Z cluster-scoped-resources/core/nodes/ip-10-0-132-117.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.766741936Z cluster-scoped-resources/core/nodes/ip-10-0-138-219.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.76689712Z cluster-scoped-resources/core/nodes/ip-10-0-140-104.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767004922Z cluster-scoped-resources/core/persistentvolumes/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767073224Z cluster-scoped-resources/core/persistentvolumes/pvc-6f1bb542-bfcb-4c3d-b024-6b18bd08bbeb.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767147006Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767205227Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767261859Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/catch-all.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767353631Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/exempt.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767429793Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/global-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767512065Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767605807Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-scheduler.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767696129Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/kube-system-service-accounts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767839953Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.767940035Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver-sar.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768046538Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7681571Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-authentication-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768244452Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768327965Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768416037Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-monitoring-metrics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768505079Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver-sar.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768588591Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768677353Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/openshift-oauth-server.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768770115Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/probes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768849177Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/service-accounts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.768939449Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-leader-election.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769022261Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-node-high.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769207386Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/flowschemas/system-nodes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769286178Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769328019Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/catch-all.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769416161Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/exempt.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769492453Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/global-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769569555Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/leader-election.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769648357Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/node-high.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769726708Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/openshift-control-plane-operators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769869642Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.769992025Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-high.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770100377Z cluster-scoped-resources/flowcontrol.apiserver.k8s.io/prioritylevelconfigurations/workload-low.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770161979Z cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77017757Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.7702095Z cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770280572Z cluster-scoped-resources/imageregistry.operator.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770287112Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770322793Z cluster-scoped-resources/imageregistry.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770400935Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770436236Z cluster-scoped-resources/imageregistry.operator.openshift.io/imagepruners/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770493317Z cluster-scoped-resources/migration.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770503827Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770552099Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/console-plugin-storage-version-migration.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770643011Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-controllerconfig-storage-version-migration.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770787804Z cluster-scoped-resources/migration.k8s.io/storageversionmigrations/machineconfiguration-machineconfigpool-storage-version-migration.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770834906Z cluster-scoped-resources/oauth.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770841255Z cluster-scoped-resources/oauth.openshift.io/oauthclients/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770898377Z cluster-scoped-resources/oauth.openshift.io/oauthclients/console.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770959018Z cluster-scoped-resources/operator.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.770966349Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77100597Z cluster-scoped-resources/operator.openshift.io/clustercsidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771099882Z cluster-scoped-resources/operator.openshift.io/consoles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771141933Z cluster-scoped-resources/operator.openshift.io/consoles/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771254926Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771294866Z cluster-scoped-resources/operator.openshift.io/csisnapshotcontrollers/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771358448Z cluster-scoped-resources/operator.openshift.io/dnses/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771403299Z cluster-scoped-resources/operator.openshift.io/dnses/default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771466691Z cluster-scoped-resources/operator.openshift.io/insightsoperators/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771508542Z cluster-scoped-resources/operator.openshift.io/insightsoperators/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771611614Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771652585Z cluster-scoped-resources/operator.openshift.io/kubeapiservers/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771709177Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771751308Z cluster-scoped-resources/operator.openshift.io/kubecontrollermanagers/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771801299Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77183914Z cluster-scoped-resources/operator.openshift.io/kubeschedulers/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771891271Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.771933122Z cluster-scoped-resources/operator.openshift.io/kubestorageversionmigrators/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772000844Z cluster-scoped-resources/operator.openshift.io/networks/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772064145Z cluster-scoped-resources/operator.openshift.io/networks/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772140647Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772183058Z cluster-scoped-resources/operator.openshift.io/openshiftapiservers/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772234239Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77227691Z cluster-scoped-resources/operator.openshift.io/openshiftcontrollermanagers/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772324231Z cluster-scoped-resources/operator.openshift.io/servicecas/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772361312Z cluster-scoped-resources/operator.openshift.io/servicecas/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772417204Z cluster-scoped-resources/operator.openshift.io/storages/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772454315Z cluster-scoped-resources/operator.openshift.io/storages/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772543267Z cluster-scoped-resources/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772550797Z cluster-scoped-resources/operators.coreos.com/olmconfigs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772585538Z cluster-scoped-resources/operators.coreos.com/olmconfigs/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77265839Z cluster-scoped-resources/operators.coreos.com/operators/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772700201Z cluster-scoped-resources/operators.coreos.com/operators/authorino-operator.kuadrant-system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772801333Z cluster-scoped-resources/operators.coreos.com/operators/dns-operator.kuadrant-system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.772895005Z cluster-scoped-resources/operators.coreos.com/operators/leader-worker-set.openshift-lws-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77307224Z cluster-scoped-resources/operators.coreos.com/operators/limitador-operator.kuadrant-system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773176092Z cluster-scoped-resources/operators.coreos.com/operators/openshift-cert-manager-operator.cert-manager-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773285375Z cluster-scoped-resources/operators.coreos.com/operators/openshift-custom-metrics-autoscaler-operator.openshift-keda.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773396208Z cluster-scoped-resources/operators.coreos.com/operators/rhcl-operator.kuadrant-system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77349222Z cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773564982Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773600353Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/aws-ebs-csi-driver-operator-clusterrolebinding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773696505Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cloud-network-config-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.773774687Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-node-tuning-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77392307Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/cluster-storage-operator-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774068404Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774185347Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/csi-snapshot-controller-runner-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774297359Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774534825Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774640328Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-ancillary-tools.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77472908Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-cluster-readers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774810042Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-group.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774913924Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-transient.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.774997396Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775110629Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775195091Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-node-identity.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775288374Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775376245Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-iptables-alerter.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775463548Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77554729Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-identity-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775630992Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-ovn-kubernetes-node-kube-rbac-proxy.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775717934Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775814486Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.775860117Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/aws-ebs-csi-driver-operator-clusterrole.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77596165Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cloud-network-config-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776060372Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/cluster-node-tuning-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776147094Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-extensions-reader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776294578Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77638328Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/console.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776470492Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/csi-snapshot-controller-operator-clusterrole.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776560234Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/helm-chartrepos-viewer.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776642476Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/metrics-daemon-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776726948Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-admission-controller-webhook.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776811271Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus-ancillary-tools.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776897422Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/multus.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.776983055Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/net-attach-def-project.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777124208Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77721161Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/network-node-identity.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777302082Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-csi-snapshot-controller-runner.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777412585Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-iptables-alerter.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777502977Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-cluster-reader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777588109Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777683421Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-kube-rbac-proxy.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777769214Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777860026Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-editor.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.777951398Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/openshift-ovn-kubernetes-udn-viewer.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77803962Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/project-helm-chartrepository-editor.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778151733Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-admin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778240785Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-basic-user.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778347738Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-storage-admin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77843425Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:openshift:aggregate-snapshots-to-view.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778520672Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/system:registry.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778620454Z cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/whereabouts-cni.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778668305Z cluster-scoped-resources/samples.operator.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778674276Z cluster-scoped-resources/samples.operator.openshift.io/configs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778717217Z cluster-scoped-resources/samples.operator.openshift.io/configs/cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778767938Z cluster-scoped-resources/snapshot.storage.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778774818Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778801349Z cluster-scoped-resources/snapshot.storage.k8s.io/volumesnapshotclasses/csi-aws-vsc.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778886721Z cluster-scoped-resources/storage.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778893361Z cluster-scoped-resources/storage.k8s.io/csidrivers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.778941192Z cluster-scoped-resources/storage.k8s.io/csidrivers/ebs.csi.aws.com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779004074Z cluster-scoped-resources/storage.k8s.io/csinodes/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779072125Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-132-117.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779161157Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-138-219.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779240679Z cluster-scoped-resources/storage.k8s.io/csinodes/ip-10-0-140-104.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779298551Z cluster-scoped-resources/storage.k8s.io/storageclasses/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779344512Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp2-csi.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779425604Z cluster-scoped-resources/storage.k8s.io/storageclasses/gp3-csi.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779483645Z host_service_logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779494885Z host_service_logs/masters/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779548417Z host_service_logs/masters/NetworkManager_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779604918Z host_service_logs/masters/crio_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.77967082Z host_service_logs/masters/kubelet_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779734961Z host_service_logs/masters/machine-config-daemon-firstboot_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779875765Z host_service_logs/masters/machine-config-daemon-host_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.779984237Z host_service_logs/masters/openvswitch_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780070059Z host_service_logs/masters/ostree-finalize-staged_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780141841Z host_service_logs/masters/ovs-configuration_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780208233Z host_service_logs/masters/ovs-vswitchd_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780283325Z host_service_logs/masters/ovsdb-server_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780346836Z host_service_logs/masters/rpm-ostreed_service.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780393917Z ingress_controllers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780400038Z ingress_controllers/default/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780408308Z ingress_controllers/default/router-default-bcd57767c-rwpfk/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780456739Z ingress_controllers/default/router-default-bcd57767c-rwpfk/haproxy.config [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780563371Z insights-data/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780575942Z istio/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780617013Z istio/aggregated-discovery-api.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.780732615Z istio/aggregated-discovery-apis.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781246408Z istio/event-filter.html [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781388851Z istio/timestamp [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781514734Z istio/cluster-scoped-resources/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781522115Z istio/cluster-scoped-resources/admissionregistration.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781530785Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781581956Z istio/cluster-scoped-resources/admissionregistration.k8s.io/mutatingwebhookconfigurations/istio-sidecar-injector-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781656548Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781701769Z istio/cluster-scoped-resources/admissionregistration.k8s.io/validatingwebhookconfigurations/istio-validator-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.78175538Z istio/cluster-scoped-resources/apiextensions.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781828502Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.781875253Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/authorizationpolicies.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.782059138Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/destinationrules.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.782813776Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/envoyfilters.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.783042051Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gatewayclasses.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.783217616Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.783600115Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/gateways.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.783769869Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/grpcroutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.784181089Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/httproutes.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.784962068Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencemodelrewrites.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785101672Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferenceobjectives.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785208044Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepoolimports.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785344258Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785497041Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/inferencepools.inference.networking.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785623024Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/peerauthentications.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785745237Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/proxyconfigs.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785846479Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/referencegrants.gateway.networking.k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.785958332Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/requestauthentications.security.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.786170278Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/serviceentries.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.786342052Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/sidecars.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.786595898Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/telemetries.telemetry.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.786777152Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/virtualservices.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787198823Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/wasmplugins.extensions.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787320795Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadentries.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787448378Z istio/cluster-scoped-resources/apiextensions.k8s.io/customresourcedefinitions/workloadgroups.networking.istio.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787591832Z istio/cluster-scoped-resources/gateway.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787601632Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787645313Z istio/cluster-scoped-resources/gateway.networking.k8s.io/gatewayclasses/openshift-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787719665Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787729295Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787769266Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.787865119Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788006172Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788096694Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788144655Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istio-reader-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788238708Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-clusterrole-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788387151Z istio/cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/istiod-gateway-controller-openshift-gateway-openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788450173Z istio/namespaces/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788461563Z istio/namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788471483Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788477093Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788524464Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788622497Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788706669Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.78875529Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788848292Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.788932774Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-3.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789017896Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/router-route-4.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789128789Z istio/namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/httproutes/scheduler-v06-pd-migration-test-kserve-route.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789248492Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789259082Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789292603Z istio/namespaces/kserve-ci-e2e-test/inference.networking.k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789365135Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789378165Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789391346Z istio/namespaces/kserve-ci-e2e-test/inference.networking.x-k8s.io/inferencepools/scheduler-v06-pd-migration-test-inference-pool.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789450037Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789459777Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789515649Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-scheduler.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789612081Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-shadow-svc.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789752324Z istio/namespaces/kserve-ci-e2e-test/networking.istio.io/destinationrules/scheduler-v06-pd-migration-test-kserve-workload-svc.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789816706Z istio/namespaces/openshift-ingress/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789874427Z istio/namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.789925098Z istio/namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.78996885Z istio/namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790017401Z istio/namespaces/openshift-ingress/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790152284Z istio/namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790233426Z istio/namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790497072Z istio/namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790691327Z istio/namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790741948Z istio/namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790789079Z istio/namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790871232Z istio/namespaces/openshift-ingress/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.790921293Z istio/namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791005745Z istio/namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791077937Z istio/namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791113027Z istio/namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.79120847Z istio/namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791264121Z istio/namespaces/openshift-ingress/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791317672Z istio/namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791714002Z istio/namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.791816535Z istio/namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792020509Z istio/namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792135902Z istio/namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792396358Z istio/namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792519201Z istio/namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792729217Z istio/namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792805388Z istio/namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792832319Z istio/namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792913491Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792923171Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.792951612Z istio/namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793041804Z istio/namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793083255Z istio/namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793161917Z istio/namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793198148Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.79328138Z istio/namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793366222Z istio/namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793414653Z istio/namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793460324Z istio/namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793506085Z istio/namespaces/openshift-ingress/networking.istio.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793516096Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793557867Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-auth-openshift-ai-inference.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793650059Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/kuadrant-openshift-ai-inference.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793768842Z istio/namespaces/openshift-ingress/networking.istio.io/envoyfilters/openshift-ai-inference-authn-ssl.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793868184Z istio/namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793915035Z istio/namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793981217Z istio/namespaces/openshift-ingress/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.793988847Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.794050819Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/istiod-openshift-gateway-94bb8fbfd-kqnw7.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.794204693Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.794213343Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.794221193Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.794271014Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.829252103Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.829325175Z istio/namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.829370136Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.829430647Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/openshift-ai-inference-openshift-default-655f7c58b4-57g8x.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.82953512Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.82954541Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.8295603Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.829602111Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.830234327Z istio/namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832347648Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832397019Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router-default-bcd57767c-rwpfk.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832507442Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832516042Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832525302Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832567603Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832678716Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832751858Z istio/namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832768338Z istio/namespaces/openshift-ingress/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832825319Z istio/namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832871561Z istio/namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832898571Z istio/namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832954173Z machine_config_ondisk/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832962183Z machine_config_termination_logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832968863Z monitoring/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.832974903Z monitoring/alertmanager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.833012504Z monitoring/alertmanager/status.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.833124457Z monitoring/alertmanager/status.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.833182378Z monitoring/prometheus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.833225509Z monitoring/prometheus/alertmanagers.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.833309941Z monitoring/prometheus/alertmanagers.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.833379303Z monitoring/prometheus/rules.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.834007338Z monitoring/prometheus/rules.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.83407476Z monitoring/prometheus/prometheus-k8s-0/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.834123171Z monitoring/prometheus/prometheus-k8s-0/active-targets.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.834952261Z monitoring/prometheus/prometheus-k8s-0/active-targets.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835005362Z monitoring/prometheus/prometheus-k8s-0/status/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835076644Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835163066Z monitoring/prometheus/prometheus-k8s-0/status/runtimeinfo.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835224138Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.83531807Z monitoring/prometheus/prometheus-k8s-0/status/tsdb.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835350891Z monitoring/prometheus/status/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835394052Z monitoring/prometheus/status/config.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835823142Z monitoring/prometheus/status/config.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835887404Z monitoring/prometheus/status/flags.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.835978976Z monitoring/prometheus/status/flags.stderr [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836165731Z namespaces/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836174771Z namespaces/cert-manager-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836178441Z namespaces/cert-manager-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836181701Z namespaces/cert-manager-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836219392Z namespaces/cert-manager-operator/coordination.k8s.io/leases/cert-manager-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836285463Z namespaces/cert-manager-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836293484Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836339055Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.83654128Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836745905Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.836888678Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837124774Z namespaces/cert-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.83736122Z namespaces/cert-manager-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837405001Z namespaces/cert-manager-operator/operators.coreos.com/installplans/install-n9nrs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837545524Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837592055Z namespaces/cert-manager-operator/operators.coreos.com/operatorconditions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837652397Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837700478Z namespaces/cert-manager-operator/operators.coreos.com/operatorgroups/openshift-cert-manager-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.837755619Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.83779579Z namespaces/cert-manager-operator/operators.coreos.com/subscriptions/openshift-cert-manager-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.839489301Z namespaces/cert-manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.839502172Z namespaces/cert-manager/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.839509282Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.839561503Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.839744827Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.839959733Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840143657Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840381533Z namespaces/cert-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840595528Z namespaces/default/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840643929Z namespaces/default/default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840717171Z namespaces/default/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840763202Z namespaces/default/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840837514Z namespaces/default/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840873295Z namespaces/default/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.840959297Z namespaces/default/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841062559Z namespaces/default/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841159012Z namespaces/default/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841196462Z namespaces/default/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841237914Z namespaces/default/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841281035Z namespaces/default/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841320686Z namespaces/default/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841400707Z namespaces/default/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841459119Z namespaces/default/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84150636Z namespaces/default/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841584442Z namespaces/default/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841660774Z namespaces/default/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841737156Z namespaces/default/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841844578Z namespaces/default/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.841930411Z namespaces/default/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842121195Z namespaces/default/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842190097Z namespaces/default/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842267719Z namespaces/default/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842388322Z namespaces/default/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842551986Z namespaces/default/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842625847Z namespaces/default/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842668378Z namespaces/default/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84274024Z namespaces/default/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842782921Z namespaces/default/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842863963Z namespaces/default/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.842911224Z namespaces/default/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843000396Z namespaces/default/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843096299Z namespaces/default/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84313352Z namespaces/default/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84317469Z namespaces/default/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843223852Z namespaces/default/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843271433Z namespaces/default/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843316184Z namespaces/default/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843323784Z namespaces/default/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843351785Z namespaces/default/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843498069Z namespaces/default/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843678613Z namespaces/default/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.843812596Z namespaces/default/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844003591Z namespaces/default/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844209616Z namespaces/default/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844244607Z namespaces/default/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844292308Z namespaces/default/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844333679Z namespaces/default/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84437835Z namespaces/kserve-ci-e2e-test/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84438939Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.8443936Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844431531Z namespaces/kserve-ci-e2e-test/coordination.k8s.io/leases/epp-kserve-ci-e2e-test-scheduler-ha-replicas-test-inference-pool.llm-d.ai.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844494803Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844502213Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844539874Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844634236Z namespaces/kserve-ci-e2e-test/gateway.networking.k8s.io/gateways/router-gateway-2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844699838Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844707448Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844751639Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844843751Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/podmonitors/kserve-llm-isvc-vllm-engine.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844896932Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.844946504Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845051786Z namespaces/kserve-ci-e2e-test/monitoring.coreos.com/servicemonitors/kserve-llm-isvc-scheduler.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845114108Z namespaces/kserve-ci-e2e-test/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845128428Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845174959Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845317382Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845490837Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84562641Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.845817785Z namespaces/kserve-ci-e2e-test/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84604274Z namespaces/kserve/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846103752Z namespaces/kserve/kserve.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846174934Z namespaces/kserve/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846217724Z namespaces/kserve/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846287166Z namespaces/kserve/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846332237Z namespaces/kserve/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846415269Z namespaces/kserve/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846602814Z namespaces/kserve/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846786328Z namespaces/kserve/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846826129Z namespaces/kserve/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846889841Z namespaces/kserve/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846929602Z namespaces/kserve/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.846964633Z namespaces/kserve/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847060335Z namespaces/kserve/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847129337Z namespaces/kserve/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847170158Z namespaces/kserve/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84724926Z namespaces/kserve/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84729095Z namespaces/kserve/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847296781Z namespaces/kserve/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847335902Z namespaces/kserve/coordination.k8s.io/leases/kserve-controller-manager-leader-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847426534Z namespaces/kserve/coordination.k8s.io/leases/llminferenceservice-kserve-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847514426Z namespaces/kserve/coordination.k8s.io/leases/odh-model-controller.opendatahub.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847631039Z namespaces/kserve/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84767064Z namespaces/kserve/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.847909056Z namespaces/kserve/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.848014618Z namespaces/kserve/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.848326256Z namespaces/kserve/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.848402008Z namespaces/kserve/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.848656494Z namespaces/kserve/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.848778877Z namespaces/kserve/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849063973Z namespaces/kserve/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849160336Z namespaces/kserve/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849186587Z namespaces/kserve/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849294059Z namespaces/kserve/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.84933752Z namespaces/kserve/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849403772Z namespaces/kserve/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849447603Z namespaces/kserve/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849531315Z namespaces/kserve/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849609147Z namespaces/kserve/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849669898Z namespaces/kserve/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849714729Z namespaces/kserve/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849762781Z namespaces/kserve/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849813592Z namespaces/kserve/monitoring.coreos.com/servicemonitors/model-serving-api-metrics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849898314Z namespaces/kserve/monitoring.coreos.com/servicemonitors/odh-model-controller-metrics-monitor.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849946325Z namespaces/kserve/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.849995746Z namespaces/kserve/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850075418Z namespaces/kserve/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850088248Z namespaces/kserve/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850130939Z namespaces/kserve/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850281593Z namespaces/kserve/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850489028Z namespaces/kserve/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850632342Z namespaces/kserve/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.850858777Z namespaces/kserve/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851116043Z namespaces/kserve/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851127334Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851169995Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/kserve-controller-manager-67485586b-v6pj6.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851260557Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851269937Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851274797Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851321848Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851439581Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851507273Z namespaces/kserve/pods/kserve-controller-manager-67485586b-v6pj6/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851553374Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851592255Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/llmisvc-controller-manager-598d4f65c9-qj6wh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851677467Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851683627Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851691527Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.851732948Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.854993288Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.85508269Z namespaces/kserve/pods/llmisvc-controller-manager-598d4f65c9-qj6wh/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.85511608Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855168582Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/model-serving-api-fd65d7d6-6txk2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855255254Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/server/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855261724Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/server/server/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855270194Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/server/server/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855326396Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/server/server/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855421618Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/server/server/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855483299Z namespaces/kserve/pods/model-serving-api-fd65d7d6-6txk2/server/server/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.85550769Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855551611Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/odh-model-controller-84b54b7f97-ldtzq.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855625243Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855637543Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855640983Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.855696285Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.887928056Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.887997908Z namespaces/kserve/pods/odh-model-controller-84b54b7f97-ldtzq/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888021009Z namespaces/kserve/pods/s3-init-zqr6x/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888102721Z namespaces/kserve/pods/s3-init-zqr6x/s3-init-zqr6x.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888186283Z namespaces/kserve/pods/s3-init-zqr6x/download-hf-model/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888192803Z namespaces/kserve/pods/s3-init-zqr6x/download-hf-model/download-hf-model/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888196313Z namespaces/kserve/pods/s3-init-zqr6x/download-hf-model/download-hf-model/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888237584Z namespaces/kserve/pods/s3-init-zqr6x/download-hf-model/download-hf-model/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888325086Z namespaces/kserve/pods/s3-init-zqr6x/download-hf-model/download-hf-model/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888385338Z namespaces/kserve/pods/s3-init-zqr6x/download-hf-model/download-hf-model/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888415068Z namespaces/kserve/pods/s3-init-zqr6x/s3-init/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888424059Z namespaces/kserve/pods/s3-init-zqr6x/s3-init/s3-init/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888450389Z namespaces/kserve/pods/s3-init-zqr6x/s3-init/s3-init/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.88847972Z namespaces/kserve/pods/s3-init-zqr6x/s3-init/s3-init/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888761667Z namespaces/kserve/pods/s3-init-zqr6x/s3-init/s3-init/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888824708Z namespaces/kserve/pods/s3-init-zqr6x/s3-init/s3-init/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.88887069Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888924251Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs-64568bcd49-grkcn.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.888996053Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889006483Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs/seaweedfs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889039354Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs/seaweedfs/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889104435Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs/seaweedfs/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.88931262Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs/seaweedfs/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889381012Z namespaces/kserve/pods/seaweedfs-64568bcd49-grkcn/seaweedfs/seaweedfs/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889421893Z namespaces/kserve/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889458604Z namespaces/kserve/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889535786Z namespaces/kserve/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889572276Z namespaces/kserve/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889654229Z namespaces/kuadrant-system/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889700179Z namespaces/kuadrant-system/kuadrant-system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889809672Z namespaces/kuadrant-system/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889854913Z namespaces/kuadrant-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889903364Z namespaces/kuadrant-system/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.889938075Z namespaces/kuadrant-system/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890046398Z namespaces/kuadrant-system/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890346235Z namespaces/kuadrant-system/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890610832Z namespaces/kuadrant-system/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890665783Z namespaces/kuadrant-system/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890707254Z namespaces/kuadrant-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890774396Z namespaces/kuadrant-system/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890817707Z namespaces/kuadrant-system/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890920219Z namespaces/kuadrant-system/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.89095114Z namespaces/kuadrant-system/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.890993911Z namespaces/kuadrant-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891100533Z namespaces/kuadrant-system/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891152655Z namespaces/kuadrant-system/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891162085Z namespaces/kuadrant-system/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891175765Z namespaces/kuadrant-system/coordination.k8s.io/leases/3745a16e.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891266988Z namespaces/kuadrant-system/coordination.k8s.io/leases/a3f98d6c.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.89135102Z namespaces/kuadrant-system/coordination.k8s.io/leases/aac3a15d.authorino.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891425781Z namespaces/kuadrant-system/coordination.k8s.io/leases/f139389e.kuadrant.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891477153Z namespaces/kuadrant-system/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891512423Z namespaces/kuadrant-system/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891691998Z namespaces/kuadrant-system/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.891809741Z namespaces/kuadrant-system/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.892232981Z namespaces/kuadrant-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.892313383Z namespaces/kuadrant-system/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.892637271Z namespaces/kuadrant-system/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.892767454Z namespaces/kuadrant-system/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.892983009Z namespaces/kuadrant-system/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893148193Z namespaces/kuadrant-system/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893192824Z namespaces/kuadrant-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893305547Z namespaces/kuadrant-system/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893335038Z namespaces/kuadrant-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893397909Z namespaces/kuadrant-system/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.89344859Z namespaces/kuadrant-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893532143Z namespaces/kuadrant-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893608274Z namespaces/kuadrant-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893645555Z namespaces/kuadrant-system/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893694136Z namespaces/kuadrant-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893759028Z namespaces/kuadrant-system/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893797679Z namespaces/kuadrant-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.89385569Z namespaces/kuadrant-system/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893870151Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.893915152Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.894114897Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.894335272Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.894464185Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.894683231Z namespaces/kuadrant-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.894919426Z namespaces/kuadrant-system/operators.coreos.com/installplans/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.894953657Z namespaces/kuadrant-system/operators.coreos.com/installplans/install-vb94d.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895281285Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895302255Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895394898Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.89547344Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895553341Z namespaces/kuadrant-system/operators.coreos.com/operatorconditions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895608853Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895646044Z namespaces/kuadrant-system/operators.coreos.com/operatorgroups/kuadrant.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895720245Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895762817Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/authorino-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.895853129Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/dns-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896015943Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/limitador-operator-stable-redhat-operators-openshift-marketplace.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896131006Z namespaces/kuadrant-system/operators.coreos.com/subscriptions/rhcl-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896210237Z namespaces/kuadrant-system/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896219478Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896263329Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino-7c77957575-nf65n.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.89632391Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896332551Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896341051Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.896399892Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899365134Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899452086Z namespaces/kuadrant-system/pods/authorino-7c77957575-nf65n/authorino/authorino/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899524848Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899579479Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/authorino-operator-6d85f6564-s4vc7.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899694442Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899702182Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899721083Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899759363Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.899963119Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900045021Z namespaces/kuadrant-system/pods/authorino-operator-6d85f6564-s4vc7/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900094892Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900142433Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/dns-operator-controller-manager-65b49595d7-c59jh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900234315Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900248826Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900261356Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900287796Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900406409Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900474051Z namespaces/kuadrant-system/pods/dns-operator-controller-manager-65b49595d7-c59jh/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900508892Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900569663Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin-85fd4b8b9d-pgv8h.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900679386Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900685756Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900689176Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900739947Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900820249Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900893171Z namespaces/kuadrant-system/pods/kuadrant-console-plugin-85fd4b8b9d-pgv8h/kuadrant-console-plugin/kuadrant-console-plugin/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900916762Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.900962503Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.901100996Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.901114227Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.901117996Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.901169378Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917195197Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917260738Z namespaces/kuadrant-system/pods/kuadrant-operator-controller-manager-bb7ddb7d7-l4dwz/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917297169Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.91735211Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador-limitador-69574b596d-8mmn4.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917444653Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917452523Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917456733Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917503624Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917593816Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917663508Z namespaces/kuadrant-system/pods/limitador-limitador-69574b596d-8mmn4/limitador/limitador/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917711369Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.91773084Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/limitador-operator-controller-manager-6f9f468797-f448h.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917841932Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917853623Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917859183Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.917887034Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918112019Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.91818013Z namespaces/kuadrant-system/pods/limitador-operator-controller-manager-6f9f468797-f448h/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918206491Z namespaces/kuadrant-system/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918251262Z namespaces/kuadrant-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918303533Z namespaces/kuadrant-system/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918335464Z namespaces/kuadrant-system/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918381995Z namespaces/kube-node-lease/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918391096Z namespaces/kube-node-lease/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918402256Z namespaces/kube-node-lease/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918447077Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-132-117.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918536849Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-138-219.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918615521Z namespaces/kube-node-lease/coordination.k8s.io/leases/ip-10-0-140-104.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918669312Z namespaces/kube-node-lease/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918679203Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918729664Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.918923779Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.919147224Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.919286617Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.919520303Z namespaces/kube-node-lease/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921271936Z namespaces/kube-public/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921287566Z namespaces/kube-public/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921292126Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921333767Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921509271Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921728557Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.921892111Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922171927Z namespaces/kube-public/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922445554Z namespaces/kube-system/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922493195Z namespaces/kube-system/kube-system.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922552587Z namespaces/kube-system/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922590937Z namespaces/kube-system/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922639429Z namespaces/kube-system/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92267548Z namespaces/kube-system/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922796842Z namespaces/kube-system/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922886465Z namespaces/kube-system/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.922967377Z namespaces/kube-system/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923006728Z namespaces/kube-system/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923074069Z namespaces/kube-system/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92311806Z namespaces/kube-system/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923164342Z namespaces/kube-system/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923243093Z namespaces/kube-system/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923282914Z namespaces/kube-system/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923328026Z namespaces/kube-system/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923403877Z namespaces/kube-system/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923446668Z namespaces/kube-system/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923468089Z namespaces/kube-system/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92351342Z namespaces/kube-system/coordination.k8s.io/leases/apiserver-t2km6a2uzyojd576ttge4c6olm.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923599552Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-cainjector-leader-election.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923688774Z namespaces/kube-system/coordination.k8s.io/leases/cert-manager-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.923769386Z namespaces/kube-system/coordination.k8s.io/leases/kube-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9239306Z namespaces/kube-system/coordination.k8s.io/leases/kube-scheduler.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924003882Z namespaces/kube-system/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924076314Z namespaces/kube-system/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924224647Z namespaces/kube-system/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924291409Z namespaces/kube-system/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924558465Z namespaces/kube-system/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924637397Z namespaces/kube-system/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.924916314Z namespaces/kube-system/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925083628Z namespaces/kube-system/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925420216Z namespaces/kube-system/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925483468Z namespaces/kube-system/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925528809Z namespaces/kube-system/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92557786Z namespaces/kube-system/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925619671Z namespaces/kube-system/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925664312Z namespaces/kube-system/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925703323Z namespaces/kube-system/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925778745Z namespaces/kube-system/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925866937Z namespaces/kube-system/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925902968Z namespaces/kube-system/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.925948259Z namespaces/kube-system/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92599234Z namespaces/kube-system/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926051861Z namespaces/kube-system/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926099343Z namespaces/kube-system/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926122363Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926172624Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926332218Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926507363Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926640366Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.926844441Z namespaces/kube-system/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927076777Z namespaces/kube-system/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927096457Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927140508Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer-bfr9n.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9272162Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92722272Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9272262Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927261251Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92762895Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927700592Z namespaces/kube-system/pods/global-pull-secret-syncer-bfr9n/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927742133Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927771903Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer-t6lw6.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927837515Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927843895Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927871526Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.927922667Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928361598Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928426699Z namespaces/kube-system/pods/global-pull-secret-syncer-t6lw6/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92846564Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928502391Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer-zwrwf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928570983Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928578823Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer/global-pull-secret-syncer/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928582223Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer/global-pull-secret-syncer/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.928635454Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer/global-pull-secret-syncer/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.929011953Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.929097326Z namespaces/kube-system/pods/global-pull-secret-syncer-zwrwf/global-pull-secret-syncer/global-pull-secret-syncer/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.929142767Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.929186638Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent-bl2jh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92927031Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9292817Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.92929259Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.929327641Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930301845Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930397007Z namespaces/kube-system/pods/konnectivity-agent-bl2jh/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930450998Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930500869Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent-n8ht4.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930566641Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930573581Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930577211Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.930612672Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931639977Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931710529Z namespaces/kube-system/pods/konnectivity-agent-n8ht4/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93174521Z namespaces/kube-system/pods/konnectivity-agent-njzrp/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931784161Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent-njzrp.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931857002Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931864113Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent/konnectivity-agent/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931871983Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent/konnectivity-agent/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.931927504Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent/konnectivity-agent/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.932861187Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent/konnectivity-agent/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.932934319Z namespaces/kube-system/pods/konnectivity-agent-njzrp/konnectivity-agent/konnectivity-agent/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.932971509Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933020511Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933118143Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/haproxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933128293Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933136504Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933183165Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933424351Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933494132Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-132-117.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933536113Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933580724Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933666826Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/haproxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933691397Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933701387Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.933743628Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934076536Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934170198Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-138-219.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934191989Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9342467Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934303502Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/haproxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934312932Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/haproxy/haproxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934316402Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/haproxy/haproxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934340793Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/haproxy/haproxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93463526Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/haproxy/haproxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934700351Z namespaces/kube-system/pods/kube-apiserver-proxy-ip-10-0-140-104.ec2.internal/haproxy/haproxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934737072Z namespaces/kube-system/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934772003Z namespaces/kube-system/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934818544Z namespaces/kube-system/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934830505Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934884926Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-authentication-reader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.934978638Z namespaces/kube-system/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93505369Z namespaces/kube-system/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935105021Z namespaces/kube-system/rbac.authorization.k8s.io/roles/extension-apiserver-authentication-reader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935159993Z namespaces/kube-system/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935202104Z namespaces/kube-system/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935247075Z namespaces/open-cluster-management-agent-addon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935259125Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935264355Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935304386Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/cluster-proxy.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935397548Z namespaces/open-cluster-management-agent-addon/coordination.k8s.io/leases/work-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93545061Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93546152Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935507781Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935663905Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935833029Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.935964512Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936203128Z namespaces/open-cluster-management-agent-addon/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936378802Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936388162Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936400013Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936448444Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936607298Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936779222Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.936913905Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937141461Z namespaces/open-cluster-management-b0136107-d2e9-40aa-b868-c21d27bac/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937355326Z namespaces/openshift-apiserver-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937406097Z namespaces/openshift-apiserver-operator/openshift-apiserver-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937473179Z namespaces/openshift-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93751445Z namespaces/openshift-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937581591Z namespaces/openshift-apiserver-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937625432Z namespaces/openshift-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937707024Z namespaces/openshift-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937778556Z namespaces/openshift-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937859158Z namespaces/openshift-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937899119Z namespaces/openshift-apiserver-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93793517Z namespaces/openshift-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.937992771Z namespaces/openshift-apiserver-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.938054563Z namespaces/openshift-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.938130345Z namespaces/openshift-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.938175926Z namespaces/openshift-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.938219327Z namespaces/openshift-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.938288509Z namespaces/openshift-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.93834205Z namespaces/openshift-apiserver-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.938384741Z namespaces/openshift-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.939317454Z namespaces/openshift-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.939412766Z namespaces/openshift-apiserver-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.939497538Z namespaces/openshift-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.939607201Z namespaces/openshift-apiserver-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.939689652Z namespaces/openshift-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.939826646Z namespaces/openshift-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940022151Z namespaces/openshift-apiserver-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940112483Z namespaces/openshift-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940155544Z namespaces/openshift-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940210585Z namespaces/openshift-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940250326Z namespaces/openshift-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940307378Z namespaces/openshift-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940333968Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.94042915Z namespaces/openshift-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940505812Z namespaces/openshift-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940560293Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940598305Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940666846Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940711307Z namespaces/openshift-apiserver-operator/monitoring.coreos.com/servicemonitors/openshift-apiserver-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940760798Z namespaces/openshift-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.94080413Z namespaces/openshift-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940906532Z namespaces/openshift-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940916792Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.940953703Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941119607Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941295122Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941461566Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941691561Z namespaces/openshift-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941888286Z namespaces/openshift-apiserver-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941928167Z namespaces/openshift-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.941965058Z namespaces/openshift-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942011219Z namespaces/openshift-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942130772Z namespaces/openshift-apiserver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942178023Z namespaces/openshift-apiserver/openshift-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942240944Z namespaces/openshift-apiserver/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942287055Z namespaces/openshift-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942333496Z namespaces/openshift-apiserver/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942369617Z namespaces/openshift-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.94245171Z namespaces/openshift-apiserver/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942525251Z namespaces/openshift-apiserver/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942607533Z namespaces/openshift-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942649414Z namespaces/openshift-apiserver/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942681605Z namespaces/openshift-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942736056Z namespaces/openshift-apiserver/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942782887Z namespaces/openshift-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.94287627Z namespaces/openshift-apiserver/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942926451Z namespaces/openshift-apiserver/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.942977782Z namespaces/openshift-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943071734Z namespaces/openshift-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943148966Z namespaces/openshift-apiserver/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943187837Z namespaces/openshift-apiserver/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943337141Z namespaces/openshift-apiserver/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943372052Z namespaces/openshift-apiserver/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943481055Z namespaces/openshift-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943569326Z namespaces/openshift-apiserver/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943647379Z namespaces/openshift-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943775182Z namespaces/openshift-apiserver/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943945436Z namespaces/openshift-apiserver/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.943997767Z namespaces/openshift-apiserver/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.944050188Z namespaces/openshift-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.94412463Z namespaces/openshift-apiserver/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.947686337Z namespaces/openshift-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.947725947Z namespaces/openshift-apiserver/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.947777349Z namespaces/openshift-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.947856101Z namespaces/openshift-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.947930263Z namespaces/openshift-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.947972744Z namespaces/openshift-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948022085Z namespaces/openshift-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948100427Z namespaces/openshift-apiserver/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948145528Z namespaces/openshift-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948185289Z namespaces/openshift-apiserver/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948200099Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.94824233Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948402114Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948586528Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948721162Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.948965397Z namespaces/openshift-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949173522Z namespaces/openshift-apiserver/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949205303Z namespaces/openshift-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949266905Z namespaces/openshift-apiserver/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949308726Z namespaces/openshift-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949528921Z namespaces/openshift-authentication-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949537082Z namespaces/openshift-authentication-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949541331Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949584823Z namespaces/openshift-authentication-operator/monitoring.coreos.com/servicemonitors/authentication-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949646614Z namespaces/openshift-authentication-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949655554Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949699455Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.949848479Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950042584Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950187587Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950401262Z namespaces/openshift-authentication-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950574937Z namespaces/openshift-authentication/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950581477Z namespaces/openshift-authentication/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950588147Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950640528Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950792662Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.950979626Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9511376Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951358656Z namespaces/openshift-authentication/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9515362Z namespaces/openshift-cloud-controller-manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95155381Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951562791Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951593331Z namespaces/openshift-cloud-controller-manager/coordination.k8s.io/leases/cloud-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951661693Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951674023Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951704504Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.951859168Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952049163Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952187596Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952415231Z namespaces/openshift-cloud-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952604746Z namespaces/openshift-cloud-credential-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952611026Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952614646Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952657027Z namespaces/openshift-cloud-credential-operator/coordination.k8s.io/leases/cloud-credential-operator-leader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952707828Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952716379Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95276415Z namespaces/openshift-cloud-credential-operator/monitoring.coreos.com/servicemonitors/cloud-credential-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952824501Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952837611Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.952882763Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953042407Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953220861Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953350164Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953575469Z namespaces/openshift-cloud-credential-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953775534Z namespaces/openshift-cloud-network-config-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953821355Z namespaces/openshift-cloud-network-config-controller/openshift-cloud-network-config-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953880467Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953922188Z namespaces/openshift-cloud-network-config-controller/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.953982809Z namespaces/openshift-cloud-network-config-controller/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954038921Z namespaces/openshift-cloud-network-config-controller/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954207715Z namespaces/openshift-cloud-network-config-controller/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954296587Z namespaces/openshift-cloud-network-config-controller/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954371969Z namespaces/openshift-cloud-network-config-controller/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95442071Z namespaces/openshift-cloud-network-config-controller/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954457081Z namespaces/openshift-cloud-network-config-controller/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954503102Z namespaces/openshift-cloud-network-config-controller/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954542013Z namespaces/openshift-cloud-network-config-controller/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954633415Z namespaces/openshift-cloud-network-config-controller/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954661676Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954703137Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954811919Z namespaces/openshift-cloud-network-config-controller/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95486194Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954874521Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.954926572Z namespaces/openshift-cloud-network-config-controller/coordination.k8s.io/leases/cloud-network-config-controller-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955018254Z namespaces/openshift-cloud-network-config-controller/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955084846Z namespaces/openshift-cloud-network-config-controller/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955198639Z namespaces/openshift-cloud-network-config-controller/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955274851Z namespaces/openshift-cloud-network-config-controller/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955356303Z namespaces/openshift-cloud-network-config-controller/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955431024Z namespaces/openshift-cloud-network-config-controller/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955510206Z namespaces/openshift-cloud-network-config-controller/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955686901Z namespaces/openshift-cloud-network-config-controller/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955791293Z namespaces/openshift-cloud-network-config-controller/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955836774Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955882605Z namespaces/openshift-cloud-network-config-controller/core/serviceaccounts/cloud-network-config-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955931037Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.955974418Z namespaces/openshift-cloud-network-config-controller/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956016199Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95608327Z namespaces/openshift-cloud-network-config-controller/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956152302Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956200693Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956278775Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956360967Z namespaces/openshift-cloud-network-config-controller/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956404428Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956446979Z namespaces/openshift-cloud-network-config-controller/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9564913Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956529781Z namespaces/openshift-cloud-network-config-controller/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956574042Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956580312Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956623203Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956769657Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.956943481Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957106815Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957348331Z namespaces/openshift-cloud-network-config-controller/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957526575Z namespaces/openshift-cloud-network-config-controller/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957566536Z namespaces/openshift-cloud-network-config-controller/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957607157Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957622587Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957656728Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/rolebindings/cloud-network-config-controller-rb.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95770034Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957746441Z namespaces/openshift-cloud-network-config-controller/rbac.authorization.k8s.io/roles/cloud-network-config-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957795192Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957834093Z namespaces/openshift-cloud-network-config-controller/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957913955Z namespaces/openshift-cluster-csi-drivers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.957957086Z namespaces/openshift-cluster-csi-drivers/openshift-cluster-csi-drivers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958022117Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958088599Z namespaces/openshift-cluster-csi-drivers/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958153Z namespaces/openshift-cluster-csi-drivers/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958202402Z namespaces/openshift-cluster-csi-drivers/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958324595Z namespaces/openshift-cluster-csi-drivers/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958398146Z namespaces/openshift-cluster-csi-drivers/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958477828Z namespaces/openshift-cluster-csi-drivers/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958520469Z namespaces/openshift-cluster-csi-drivers/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95856421Z namespaces/openshift-cluster-csi-drivers/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958611282Z namespaces/openshift-cluster-csi-drivers/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958653292Z namespaces/openshift-cluster-csi-drivers/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958738325Z namespaces/openshift-cluster-csi-drivers/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958785236Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958829797Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958927599Z namespaces/openshift-cluster-csi-drivers/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95897589Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.958983771Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959016601Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/ebs-csi-aws-com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959139474Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-attacher-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959224186Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-resizer-ebs-csi-aws-com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.95935332Z namespaces/openshift-cluster-csi-drivers/coordination.k8s.io/leases/external-snapshotter-leader-ebs-csi-aws-com.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959448972Z namespaces/openshift-cluster-csi-drivers/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959493873Z namespaces/openshift-cluster-csi-drivers/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959588635Z namespaces/openshift-cluster-csi-drivers/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959663787Z namespaces/openshift-cluster-csi-drivers/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.959991235Z namespaces/openshift-cluster-csi-drivers/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960095538Z namespaces/openshift-cluster-csi-drivers/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960318203Z namespaces/openshift-cluster-csi-drivers/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960479547Z namespaces/openshift-cluster-csi-drivers/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96058707Z namespaces/openshift-cluster-csi-drivers/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960627111Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960672421Z namespaces/openshift-cluster-csi-drivers/core/serviceaccounts/aws-ebs-csi-driver-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960720493Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960767594Z namespaces/openshift-cluster-csi-drivers/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960806795Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960847186Z namespaces/openshift-cluster-csi-drivers/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960890077Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.960932348Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96100986Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961122512Z namespaces/openshift-cluster-csi-drivers/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961171714Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961208485Z namespaces/openshift-cluster-csi-drivers/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961248215Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961297077Z namespaces/openshift-cluster-csi-drivers/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961361098Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961372629Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96142204Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961575843Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961746558Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.961887771Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962132877Z namespaces/openshift-cluster-csi-drivers/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962327142Z namespaces/openshift-cluster-csi-drivers/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962336182Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962375863Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/aws-ebs-csi-driver-node-m8rv9.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962477535Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-driver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962488036Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-driver/csi-driver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962492296Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-driver/csi-driver/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962540457Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962634269Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962700761Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962738082Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-liveness-probe/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962745632Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962752652Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962796093Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962894215Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.962967847Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963131911Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-node-driver-registrar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963144681Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963151752Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963192683Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963287665Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963354517Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-m8rv9/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963404308Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963451089Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/aws-ebs-csi-driver-node-n5hwh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963536511Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-driver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963544921Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-driver/csi-driver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963553531Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-driver/csi-driver/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963598563Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963689185Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963759047Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963797937Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-liveness-probe/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963806928Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963812918Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963850089Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.963936461Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964007413Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964066784Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-node-driver-registrar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964077554Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964087455Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964129845Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964223358Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964289149Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-n5hwh/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96433979Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964388032Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/aws-ebs-csi-driver-node-wrnpm.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964468854Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-driver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964485624Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-driver/csi-driver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964490834Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-driver/csi-driver/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964533855Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-driver/csi-driver/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964630277Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-driver/csi-driver/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964696729Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-driver/csi-driver/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96474158Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-liveness-probe/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96474796Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-liveness-probe/csi-liveness-probe/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964757921Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-liveness-probe/csi-liveness-probe/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964791801Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-liveness-probe/csi-liveness-probe/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964881004Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-liveness-probe/csi-liveness-probe/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964946905Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-liveness-probe/csi-liveness-probe/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964993236Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-node-driver-registrar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.964999617Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-node-driver-registrar/csi-node-driver-registrar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965005427Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-node-driver-registrar/csi-node-driver-registrar/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965079819Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-node-driver-registrar/csi-node-driver-registrar/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965171391Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965237672Z namespaces/openshift-cluster-csi-drivers/pods/aws-ebs-csi-driver-node-wrnpm/csi-node-driver-registrar/csi-node-driver-registrar/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965286053Z namespaces/openshift-cluster-csi-drivers/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965327474Z namespaces/openshift-cluster-csi-drivers/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965382436Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965388656Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965433777Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/rolebindings/aws-ebs-csi-driver-operator-rolebinding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965497729Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96553729Z namespaces/openshift-cluster-csi-drivers/rbac.authorization.k8s.io/roles/aws-ebs-csi-driver-operator-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965590891Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965629142Z namespaces/openshift-cluster-csi-drivers/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965688723Z namespaces/openshift-cluster-machine-approver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965694963Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965701674Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965743875Z namespaces/openshift-cluster-machine-approver/coordination.k8s.io/leases/cluster-machine-approver-leader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965804986Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965811136Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965848717Z namespaces/openshift-cluster-machine-approver/monitoring.coreos.com/servicemonitors/cluster-machine-approver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965903008Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.965915839Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9659598Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.966137704Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.966310268Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.966456292Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.966691808Z namespaces/openshift-cluster-machine-approver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.966894613Z namespaces/openshift-cluster-node-tuning-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.966944724Z namespaces/openshift-cluster-node-tuning-operator/openshift-cluster-node-tuning-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967012326Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967078857Z namespaces/openshift-cluster-node-tuning-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967148009Z namespaces/openshift-cluster-node-tuning-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96718879Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967312953Z namespaces/openshift-cluster-node-tuning-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967390435Z namespaces/openshift-cluster-node-tuning-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967466166Z namespaces/openshift-cluster-node-tuning-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967501347Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967553339Z namespaces/openshift-cluster-node-tuning-operator/apps/daemonsets/tuned.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967691002Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967730973Z namespaces/openshift-cluster-node-tuning-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967770234Z namespaces/openshift-cluster-node-tuning-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967808815Z namespaces/openshift-cluster-node-tuning-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967883147Z namespaces/openshift-cluster-node-tuning-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967926988Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.967966078Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.968071081Z namespaces/openshift-cluster-node-tuning-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.968103642Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.968116912Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.968167694Z namespaces/openshift-cluster-node-tuning-operator/coordination.k8s.io/leases/node-tuning-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.968248336Z namespaces/openshift-cluster-node-tuning-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.968291117Z namespaces/openshift-cluster-node-tuning-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.969154418Z namespaces/openshift-cluster-node-tuning-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96925914Z namespaces/openshift-cluster-node-tuning-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.969393243Z namespaces/openshift-cluster-node-tuning-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.969470325Z namespaces/openshift-cluster-node-tuning-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.96967576Z namespaces/openshift-cluster-node-tuning-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.969849404Z namespaces/openshift-cluster-node-tuning-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.969988348Z namespaces/openshift-cluster-node-tuning-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97008756Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970157142Z namespaces/openshift-cluster-node-tuning-operator/core/serviceaccounts/cluster-node-tuning-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970212573Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970247594Z namespaces/openshift-cluster-node-tuning-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970307485Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970343866Z namespaces/openshift-cluster-node-tuning-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970400198Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970446789Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970537161Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970615613Z namespaces/openshift-cluster-node-tuning-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970673554Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970719945Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970780557Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970828148Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/prometheusrules/node-tuning-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97091662Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.970962241Z namespaces/openshift-cluster-node-tuning-operator/monitoring.coreos.com/servicemonitors/node-tuning-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971018913Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971084004Z namespaces/openshift-cluster-node-tuning-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971135966Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971153416Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971246838Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971438533Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971628107Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.971772501Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972044377Z namespaces/openshift-cluster-node-tuning-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972286463Z namespaces/openshift-cluster-node-tuning-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972304404Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972367055Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned-4x2pr.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972468988Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972475298Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned/tuned/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972488768Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned/tuned/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972535989Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned/tuned/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972653582Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972723624Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-4x2pr/tuned/tuned/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972816376Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972867428Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned-tml2k.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97294861Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97295479Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned/tuned/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97295821Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned/tuned/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.972994481Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned/tuned/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973137084Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973235817Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-tml2k/tuned/tuned/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973271837Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973324409Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned-wjlv6.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973400401Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973406441Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned/tuned/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973409791Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned/tuned/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973446811Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned/tuned/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973559144Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned/tuned/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973619196Z namespaces/openshift-cluster-node-tuning-operator/pods/tuned-wjlv6/tuned/tuned/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973651916Z namespaces/openshift-cluster-node-tuning-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973690908Z namespaces/openshift-cluster-node-tuning-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973743939Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97378277Z namespaces/openshift-cluster-node-tuning-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973826121Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973894142Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.973941194Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-132-117.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974049666Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-138-219.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974148339Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/profiles/ip-10-0-140-104.ec2.internal.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97419474Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974233941Z namespaces/openshift-cluster-node-tuning-operator/tuned.openshift.io/tuneds/default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974348633Z namespaces/openshift-cluster-samples-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974390214Z namespaces/openshift-cluster-samples-operator/openshift-cluster-samples-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974456606Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974505247Z namespaces/openshift-cluster-samples-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974557349Z namespaces/openshift-cluster-samples-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974600859Z namespaces/openshift-cluster-samples-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974682561Z namespaces/openshift-cluster-samples-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974806684Z namespaces/openshift-cluster-samples-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974918447Z namespaces/openshift-cluster-samples-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.974972048Z namespaces/openshift-cluster-samples-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97501689Z namespaces/openshift-cluster-samples-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975085931Z namespaces/openshift-cluster-samples-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975128452Z namespaces/openshift-cluster-samples-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975211624Z namespaces/openshift-cluster-samples-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975263886Z namespaces/openshift-cluster-samples-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975308297Z namespaces/openshift-cluster-samples-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975381778Z namespaces/openshift-cluster-samples-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97546612Z namespaces/openshift-cluster-samples-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975504101Z namespaces/openshift-cluster-samples-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975677836Z namespaces/openshift-cluster-samples-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975761138Z namespaces/openshift-cluster-samples-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975891711Z namespaces/openshift-cluster-samples-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.975966763Z namespaces/openshift-cluster-samples-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976107186Z namespaces/openshift-cluster-samples-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976234709Z namespaces/openshift-cluster-samples-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976429324Z namespaces/openshift-cluster-samples-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976481705Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976526816Z namespaces/openshift-cluster-samples-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976585198Z namespaces/openshift-cluster-samples-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976629969Z namespaces/openshift-cluster-samples-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9766933Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976734061Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976812173Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976888135Z namespaces/openshift-cluster-samples-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976934866Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.976975807Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977040439Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97709901Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/prometheusrules/samples-operator-alerts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977168032Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977213663Z namespaces/openshift-cluster-samples-operator/monitoring.coreos.com/servicemonitors/cluster-samples-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977257644Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977297795Z namespaces/openshift-cluster-samples-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977340706Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977347806Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977386317Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977523641Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977706825Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.977840458Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978091074Z namespaces/openshift-cluster-samples-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978277399Z namespaces/openshift-cluster-samples-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978288919Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97832531Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-9ddc5ff76-9lfqm.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978412582Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-watch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978419232Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-watch/cluster-samples-operator-watch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978422592Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978463753Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978556915Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978619337Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator-watch/cluster-samples-operator-watch/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978663828Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978671128Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator/cluster-samples-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978676538Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator/cluster-samples-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.978717959Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator/cluster-samples-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.979480968Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator/cluster-samples-operator/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.97955417Z namespaces/openshift-cluster-samples-operator/pods/cluster-samples-operator-9ddc5ff76-9lfqm/cluster-samples-operator/cluster-samples-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.979594971Z namespaces/openshift-cluster-samples-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.979662492Z namespaces/openshift-cluster-samples-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.979713363Z namespaces/openshift-cluster-samples-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.979822276Z namespaces/openshift-cluster-samples-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.979940039Z namespaces/openshift-cluster-storage-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98000323Z namespaces/openshift-cluster-storage-operator/openshift-cluster-storage-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980098943Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980149624Z namespaces/openshift-cluster-storage-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980233266Z namespaces/openshift-cluster-storage-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980324918Z namespaces/openshift-cluster-storage-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980422091Z namespaces/openshift-cluster-storage-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980524753Z namespaces/openshift-cluster-storage-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980620486Z namespaces/openshift-cluster-storage-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980661967Z namespaces/openshift-cluster-storage-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980704298Z namespaces/openshift-cluster-storage-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980749229Z namespaces/openshift-cluster-storage-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98082159Z namespaces/openshift-cluster-storage-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980907003Z namespaces/openshift-cluster-storage-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.980981514Z namespaces/openshift-cluster-storage-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981069916Z namespaces/openshift-cluster-storage-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981159879Z namespaces/openshift-cluster-storage-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981343173Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981355314Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981401304Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/data-source-validator-leader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981489547Z namespaces/openshift-cluster-storage-operator/coordination.k8s.io/leases/snapshot-controller-leader.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981558298Z namespaces/openshift-cluster-storage-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981602389Z namespaces/openshift-cluster-storage-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981708412Z namespaces/openshift-cluster-storage-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.981789984Z namespaces/openshift-cluster-storage-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982078371Z namespaces/openshift-cluster-storage-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982151893Z namespaces/openshift-cluster-storage-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982263025Z namespaces/openshift-cluster-storage-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98243298Z namespaces/openshift-cluster-storage-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982624044Z namespaces/openshift-cluster-storage-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982700136Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982744617Z namespaces/openshift-cluster-storage-operator/core/serviceaccounts/csi-snapshot-controller-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982809409Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982839459Z namespaces/openshift-cluster-storage-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982917681Z namespaces/openshift-cluster-storage-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.982952832Z namespaces/openshift-cluster-storage-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983009514Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983073975Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983152447Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983228689Z namespaces/openshift-cluster-storage-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98329017Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983334971Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983384123Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983416593Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/prometheusrules/prometheus.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983510636Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983548507Z namespaces/openshift-cluster-storage-operator/monitoring.coreos.com/servicemonitors/cluster-storage-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983607998Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983652419Z namespaces/openshift-cluster-storage-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983735991Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983742111Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983780592Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.983924086Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984128601Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984258334Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98449259Z namespaces/openshift-cluster-storage-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984670034Z namespaces/openshift-cluster-storage-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984680874Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984706325Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator-69d9f679db-gd5rv.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984785157Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984796567Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator/volume-data-source-validator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984800887Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator/volume-data-source-validator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984816907Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator/volume-data-source-validator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9849165Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator/volume-data-source-validator/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.984981402Z namespaces/openshift-cluster-storage-operator/pods/volume-data-source-validator-69d9f679db-gd5rv/volume-data-source-validator/volume-data-source-validator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985012742Z namespaces/openshift-cluster-storage-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985076064Z namespaces/openshift-cluster-storage-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985124055Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985134605Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985175136Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/rolebindings/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985231748Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985260808Z namespaces/openshift-cluster-storage-operator/rbac.authorization.k8s.io/roles/csi-snapshot-controller-operator-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98531501Z namespaces/openshift-cluster-storage-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9853541Z namespaces/openshift-cluster-storage-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985428052Z namespaces/openshift-cluster-version/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985472403Z namespaces/openshift-cluster-version/openshift-cluster-version.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985533985Z namespaces/openshift-cluster-version/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985570006Z namespaces/openshift-cluster-version/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985627417Z namespaces/openshift-cluster-version/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985666698Z namespaces/openshift-cluster-version/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98574346Z namespaces/openshift-cluster-version/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985815282Z namespaces/openshift-cluster-version/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985888924Z namespaces/openshift-cluster-version/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985930574Z namespaces/openshift-cluster-version/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.985972235Z namespaces/openshift-cluster-version/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986044347Z namespaces/openshift-cluster-version/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986091298Z namespaces/openshift-cluster-version/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986195851Z namespaces/openshift-cluster-version/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986266673Z namespaces/openshift-cluster-version/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986323724Z namespaces/openshift-cluster-version/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986418626Z namespaces/openshift-cluster-version/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986460347Z namespaces/openshift-cluster-version/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986466338Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986501978Z namespaces/openshift-cluster-version/coordination.k8s.io/leases/version.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98658929Z namespaces/openshift-cluster-version/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986626251Z namespaces/openshift-cluster-version/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986723184Z namespaces/openshift-cluster-version/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986800546Z namespaces/openshift-cluster-version/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.986906288Z namespaces/openshift-cluster-version/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98697789Z namespaces/openshift-cluster-version/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987077502Z namespaces/openshift-cluster-version/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987206685Z namespaces/openshift-cluster-version/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98739481Z namespaces/openshift-cluster-version/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987450691Z namespaces/openshift-cluster-version/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987501292Z namespaces/openshift-cluster-version/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987545904Z namespaces/openshift-cluster-version/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987584635Z namespaces/openshift-cluster-version/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987642106Z namespaces/openshift-cluster-version/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987684247Z namespaces/openshift-cluster-version/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987759739Z namespaces/openshift-cluster-version/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987835711Z namespaces/openshift-cluster-version/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987877602Z namespaces/openshift-cluster-version/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987918303Z namespaces/openshift-cluster-version/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.987961464Z namespaces/openshift-cluster-version/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988004355Z namespaces/openshift-cluster-version/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988075186Z namespaces/openshift-cluster-version/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988090537Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988137988Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988281131Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988454836Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988599799Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988819395Z namespaces/openshift-cluster-version/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.988995489Z namespaces/openshift-cluster-version/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98904033Z namespaces/openshift-cluster-version/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989117402Z namespaces/openshift-cluster-version/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989157763Z namespaces/openshift-cluster-version/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989243715Z namespaces/openshift-config-managed/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989275726Z namespaces/openshift-config-managed/openshift-config-managed.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989343037Z namespaces/openshift-config-managed/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989373628Z namespaces/openshift-config-managed/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98943285Z namespaces/openshift-config-managed/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989475331Z namespaces/openshift-config-managed/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989549662Z namespaces/openshift-config-managed/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989621354Z namespaces/openshift-config-managed/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989697386Z namespaces/openshift-config-managed/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989799518Z namespaces/openshift-config-managed/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.98984232Z namespaces/openshift-config-managed/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989903341Z namespaces/openshift-config-managed/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.989945382Z namespaces/openshift-config-managed/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.990019504Z namespaces/openshift-config-managed/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.990094506Z namespaces/openshift-config-managed/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.990126696Z namespaces/openshift-config-managed/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.990207588Z namespaces/openshift-config-managed/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.9902788Z namespaces/openshift-config-managed/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.990321531Z namespaces/openshift-config-managed/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.993502788Z namespaces/openshift-config-managed/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.99357615Z namespaces/openshift-config-managed/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.993666872Z namespaces/openshift-config-managed/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.993747234Z namespaces/openshift-config-managed/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.993845766Z namespaces/openshift-config-managed/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994048931Z namespaces/openshift-config-managed/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994163574Z namespaces/openshift-config-managed/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994204925Z namespaces/openshift-config-managed/core/configmaps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994252346Z namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994332248Z namespaces/openshift-config-managed/core/configmaps/openshift-network-features.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.99439371Z namespaces/openshift-config-managed/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994450521Z namespaces/openshift-config-managed/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994494102Z namespaces/openshift-config-managed/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994551444Z namespaces/openshift-config-managed/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994617715Z namespaces/openshift-config-managed/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994683507Z namespaces/openshift-config-managed/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994785609Z namespaces/openshift-config-managed/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994869991Z namespaces/openshift-config-managed/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994929493Z namespaces/openshift-config-managed/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.994977174Z namespaces/openshift-config-managed/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995046075Z namespaces/openshift-config-managed/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995090237Z namespaces/openshift-config-managed/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995141318Z namespaces/openshift-config-managed/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995154138Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995200169Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995423055Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995611059Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995748113Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.995979398Z namespaces/openshift-config-managed/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996202724Z namespaces/openshift-config-managed/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996237775Z namespaces/openshift-config-managed/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996300356Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996306626Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996355187Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/rolebindings/openshift-network-public-role-binding.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996410899Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996434939Z namespaces/openshift-config-managed/rbac.authorization.k8s.io/roles/openshift-network-public-role.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996489031Z namespaces/openshift-config-managed/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996526731Z namespaces/openshift-config-managed/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996616694Z namespaces/openshift-config-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996626384Z namespaces/openshift-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996631964Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996665555Z namespaces/openshift-config-operator/monitoring.coreos.com/servicemonitors/config-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996723076Z namespaces/openshift-config-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996737027Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996780248Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.996924771Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997120406Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997250799Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997489225Z namespaces/openshift-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.99769177Z namespaces/openshift-config/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997736971Z namespaces/openshift-config/openshift-config.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997882944Z namespaces/openshift-config/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997919615Z namespaces/openshift-config/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.997997527Z namespaces/openshift-config/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998052919Z namespaces/openshift-config/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998131721Z namespaces/openshift-config/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998206492Z namespaces/openshift-config/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998281424Z namespaces/openshift-config/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998329645Z namespaces/openshift-config/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998360306Z namespaces/openshift-config/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998408987Z namespaces/openshift-config/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998448948Z namespaces/openshift-config/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.99852733Z namespaces/openshift-config/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998564241Z namespaces/openshift-config/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998590952Z namespaces/openshift-config/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998673494Z namespaces/openshift-config/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998715305Z namespaces/openshift-config/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998754085Z namespaces/openshift-config/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.998876339Z namespaces/openshift-config/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.99894893Z namespaces/openshift-config/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999044703Z namespaces/openshift-config/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999121574Z namespaces/openshift-config/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999195986Z namespaces/openshift-config/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999366751Z namespaces/openshift-config/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999547285Z namespaces/openshift-config/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999618006Z namespaces/openshift-config/core/secrets/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999668628Z namespaces/openshift-config/core/secrets/pull-secret.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999722589Z namespaces/openshift-config/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.99977368Z namespaces/openshift-config/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999813571Z namespaces/openshift-config/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999857082Z namespaces/openshift-config/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999900413Z namespaces/openshift-config/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:23.999946514Z namespaces/openshift-config/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000014256Z namespaces/openshift-config/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000115809Z namespaces/openshift-config/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00016711Z namespaces/openshift-config/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000206961Z namespaces/openshift-config/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000250972Z namespaces/openshift-config/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000298143Z namespaces/openshift-config/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000347914Z namespaces/openshift-config/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000359384Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000405846Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000548919Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000732033Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.000862347Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001124743Z namespaces/openshift-config/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001299887Z namespaces/openshift-config/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001323338Z namespaces/openshift-config/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001378679Z namespaces/openshift-config/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00141641Z namespaces/openshift-config/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001507492Z namespaces/openshift-console-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001547893Z namespaces/openshift-console-operator/openshift-console-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001682597Z namespaces/openshift-console-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001719428Z namespaces/openshift-console-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001770589Z namespaces/openshift-console-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00180692Z namespaces/openshift-console-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001878492Z namespaces/openshift-console-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.001995714Z namespaces/openshift-console-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002130187Z namespaces/openshift-console-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002175079Z namespaces/openshift-console-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00222092Z namespaces/openshift-console-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002261351Z namespaces/openshift-console-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002293671Z namespaces/openshift-console-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002370173Z namespaces/openshift-console-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002415754Z namespaces/openshift-console-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002444705Z namespaces/openshift-console-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002525267Z namespaces/openshift-console-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002562708Z namespaces/openshift-console-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002570198Z namespaces/openshift-console-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002609819Z namespaces/openshift-console-operator/coordination.k8s.io/leases/console-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002673231Z namespaces/openshift-console-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.002715662Z namespaces/openshift-console-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.003314736Z namespaces/openshift-console-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.003385118Z namespaces/openshift-console-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.003615754Z namespaces/openshift-console-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.003691975Z namespaces/openshift-console-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.003809988Z namespaces/openshift-console-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.003954872Z namespaces/openshift-console-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004265859Z namespaces/openshift-console-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004333901Z namespaces/openshift-console-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004383862Z namespaces/openshift-console-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004441604Z namespaces/openshift-console-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004486915Z namespaces/openshift-console-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004532776Z namespaces/openshift-console-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004583957Z namespaces/openshift-console-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004654739Z namespaces/openshift-console-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004730591Z namespaces/openshift-console-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004780772Z namespaces/openshift-console-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004824083Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004876244Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004916615Z namespaces/openshift-console-operator/monitoring.coreos.com/prometheusrules/cluster-monitoring-prometheus-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.004972816Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005004637Z namespaces/openshift-console-operator/monitoring.coreos.com/servicemonitors/console-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005077259Z namespaces/openshift-console-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00511709Z namespaces/openshift-console-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005163711Z namespaces/openshift-console-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005174532Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005222853Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005698684Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.005870328Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006001412Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006283858Z namespaces/openshift-console-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006467753Z namespaces/openshift-console-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006483383Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006527124Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/console-operator-7fdbdf9c99-r5d2x.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006670418Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/console-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006677638Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/console-operator/console-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006685828Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/console-operator/console-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.006735489Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/console-operator/console-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007256512Z namespaces/openshift-console-operator/pods/console-operator-7fdbdf9c99-r5d2x/console-operator/console-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007284593Z namespaces/openshift-console-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007344134Z namespaces/openshift-console-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007404606Z namespaces/openshift-console-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007439666Z namespaces/openshift-console-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007497488Z namespaces/openshift-console-user-settings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007506788Z namespaces/openshift-console-user-settings/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007511358Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00759358Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007746454Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.007960779Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.008170554Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00840516Z namespaces/openshift-console-user-settings/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.008601284Z namespaces/openshift-console/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.008648106Z namespaces/openshift-console/openshift-console.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.008779159Z namespaces/openshift-console/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00882869Z namespaces/openshift-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.008885151Z namespaces/openshift-console/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.008924263Z namespaces/openshift-console/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009004864Z namespaces/openshift-console/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009196109Z namespaces/openshift-console/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009462676Z namespaces/openshift-console/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009512827Z namespaces/openshift-console/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009550498Z namespaces/openshift-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009604779Z namespaces/openshift-console/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.00964085Z namespaces/openshift-console/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009722172Z namespaces/openshift-console/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009783033Z namespaces/openshift-console/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009844755Z namespaces/openshift-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.009951597Z namespaces/openshift-console/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.010007929Z namespaces/openshift-console/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01008028Z namespaces/openshift-console/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.010662445Z namespaces/openshift-console/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.010745977Z namespaces/openshift-console/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.010991013Z namespaces/openshift-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011084605Z namespaces/openshift-console/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011247969Z namespaces/openshift-console/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011394292Z namespaces/openshift-console/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011657609Z namespaces/openshift-console/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011745651Z namespaces/openshift-console/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011779712Z namespaces/openshift-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011853714Z namespaces/openshift-console/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011883304Z namespaces/openshift-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011948986Z namespaces/openshift-console/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.011978117Z namespaces/openshift-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012087729Z namespaces/openshift-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012160611Z namespaces/openshift-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012212672Z namespaces/openshift-console/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012260793Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012314385Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012356126Z namespaces/openshift-console/monitoring.coreos.com/servicemonitors/console.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012422447Z namespaces/openshift-console/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012475698Z namespaces/openshift-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012657043Z namespaces/openshift-console/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012663433Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012708754Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.012850168Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013019292Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013187916Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01336559Z namespaces/openshift-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013562495Z namespaces/openshift-console/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013569145Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013601496Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console-8dbc4859-4xgml.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013701438Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013709439Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console/console/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013713948Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console/console/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013742259Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console/console/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013879303Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console/console/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013944574Z namespaces/openshift-console/pods/console-8dbc4859-4xgml/console/console/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.013975315Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014018246Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/downloads-5499b6fb8f-njh65.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014167779Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/download-server/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01418011Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/download-server/download-server/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01418593Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/download-server/download-server/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014212511Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/download-server/download-server/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01459523Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/download-server/download-server/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014658551Z namespaces/openshift-console/pods/downloads-5499b6fb8f-njh65/download-server/download-server/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014686992Z namespaces/openshift-console/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014726103Z namespaces/openshift-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014794495Z namespaces/openshift-console/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014835256Z namespaces/openshift-console/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014939488Z namespaces/openshift-controller-manager-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.014987179Z namespaces/openshift-controller-manager-operator/openshift-controller-manager-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015066881Z namespaces/openshift-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015110802Z namespaces/openshift-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015167454Z namespaces/openshift-controller-manager-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015195624Z namespaces/openshift-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015275897Z namespaces/openshift-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015348928Z namespaces/openshift-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01542477Z namespaces/openshift-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015479562Z namespaces/openshift-controller-manager-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015518802Z namespaces/openshift-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015573354Z namespaces/openshift-controller-manager-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015615415Z namespaces/openshift-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015691196Z namespaces/openshift-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015749468Z namespaces/openshift-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015780919Z namespaces/openshift-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015856231Z namespaces/openshift-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015932042Z namespaces/openshift-controller-manager-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.015973483Z namespaces/openshift-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016103717Z namespaces/openshift-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016176848Z namespaces/openshift-controller-manager-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01624929Z namespaces/openshift-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016321752Z namespaces/openshift-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016397064Z namespaces/openshift-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016508186Z namespaces/openshift-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01665269Z namespaces/openshift-controller-manager-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016702961Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016743562Z namespaces/openshift-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016802474Z namespaces/openshift-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016845755Z namespaces/openshift-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016890786Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.016933917Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017011239Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017109621Z namespaces/openshift-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017149922Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017190303Z namespaces/openshift-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017221164Z namespaces/openshift-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017266165Z namespaces/openshift-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017310016Z namespaces/openshift-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017316296Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017352797Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01748944Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017665104Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017796358Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.017978402Z namespaces/openshift-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018191967Z namespaces/openshift-controller-manager-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018210358Z namespaces/openshift-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018271329Z namespaces/openshift-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01831231Z namespaces/openshift-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018386812Z namespaces/openshift-controller-manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018434523Z namespaces/openshift-controller-manager/openshift-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018479224Z namespaces/openshift-controller-manager/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018509965Z namespaces/openshift-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018572886Z namespaces/openshift-controller-manager/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018603697Z namespaces/openshift-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018686129Z namespaces/openshift-controller-manager/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018761641Z namespaces/openshift-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018834673Z namespaces/openshift-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018902014Z namespaces/openshift-controller-manager/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018942606Z namespaces/openshift-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.018987876Z namespaces/openshift-controller-manager/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019017527Z namespaces/openshift-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01912306Z namespaces/openshift-controller-manager/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019179221Z namespaces/openshift-controller-manager/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019218012Z namespaces/openshift-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019295504Z namespaces/openshift-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019342625Z namespaces/openshift-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019378366Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019417817Z namespaces/openshift-controller-manager/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019492279Z namespaces/openshift-controller-manager/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.01953159Z namespaces/openshift-controller-manager/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019633972Z namespaces/openshift-controller-manager/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019707254Z namespaces/openshift-controller-manager/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019785466Z namespaces/openshift-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019856968Z namespaces/openshift-controller-manager/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.019931149Z namespaces/openshift-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020068943Z namespaces/openshift-controller-manager/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020226026Z namespaces/openshift-controller-manager/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020402981Z namespaces/openshift-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020433752Z namespaces/openshift-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020485873Z namespaces/openshift-controller-manager/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020515604Z namespaces/openshift-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020565015Z namespaces/openshift-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020595026Z namespaces/openshift-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020675428Z namespaces/openshift-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020750809Z namespaces/openshift-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02079152Z namespaces/openshift-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020830951Z namespaces/openshift-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020873222Z namespaces/openshift-controller-manager/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020911393Z namespaces/openshift-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020952734Z namespaces/openshift-controller-manager/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020959024Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.020995555Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.021250322Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.021424326Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.021557809Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.021761244Z namespaces/openshift-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.021945208Z namespaces/openshift-controller-manager/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.021981549Z namespaces/openshift-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022044721Z namespaces/openshift-controller-manager/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022135333Z namespaces/openshift-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022200665Z namespaces/openshift-dns-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022249776Z namespaces/openshift-dns-operator/openshift-dns-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022334038Z namespaces/openshift-dns-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022373559Z namespaces/openshift-dns-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02243867Z namespaces/openshift-dns-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022477461Z namespaces/openshift-dns-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022559653Z namespaces/openshift-dns-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022637195Z namespaces/openshift-dns-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022714707Z namespaces/openshift-dns-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022755678Z namespaces/openshift-dns-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022795729Z namespaces/openshift-dns-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022865131Z namespaces/openshift-dns-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022909662Z namespaces/openshift-dns-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.022993374Z namespaces/openshift-dns-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023081986Z namespaces/openshift-dns-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023127657Z namespaces/openshift-dns-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023217209Z namespaces/openshift-dns-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023282091Z namespaces/openshift-dns-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023323612Z namespaces/openshift-dns-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023425954Z namespaces/openshift-dns-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023505206Z namespaces/openshift-dns-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023586348Z namespaces/openshift-dns-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02366303Z namespaces/openshift-dns-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023740622Z namespaces/openshift-dns-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.023904266Z namespaces/openshift-dns-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024111551Z namespaces/openshift-dns-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024169152Z namespaces/openshift-dns-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024215794Z namespaces/openshift-dns-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024261475Z namespaces/openshift-dns-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024306576Z namespaces/openshift-dns-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024357487Z namespaces/openshift-dns-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024394508Z namespaces/openshift-dns-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02447662Z namespaces/openshift-dns-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024555342Z namespaces/openshift-dns-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024597763Z namespaces/openshift-dns-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024637094Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024687025Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024722456Z namespaces/openshift-dns-operator/monitoring.coreos.com/prometheusrules/dns.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024779457Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024826988Z namespaces/openshift-dns-operator/monitoring.coreos.com/servicemonitors/dns-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024877799Z namespaces/openshift-dns-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024941921Z namespaces/openshift-dns-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.024987462Z namespaces/openshift-dns-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025001453Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025086204Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025239528Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025423233Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025561866Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02574562Z namespaces/openshift-dns-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025927255Z namespaces/openshift-dns-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.025956946Z namespaces/openshift-dns-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026010957Z namespaces/openshift-dns-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026081129Z namespaces/openshift-dns-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026160571Z namespaces/openshift-dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026206522Z namespaces/openshift-dns/openshift-dns.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026269873Z namespaces/openshift-dns/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026311044Z namespaces/openshift-dns/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026437007Z namespaces/openshift-dns/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026476708Z namespaces/openshift-dns/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026612741Z namespaces/openshift-dns/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026687063Z namespaces/openshift-dns/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026765475Z namespaces/openshift-dns/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026817946Z namespaces/openshift-dns/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026876008Z namespaces/openshift-dns/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026934769Z namespaces/openshift-dns/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.026982781Z namespaces/openshift-dns/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027087013Z namespaces/openshift-dns/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027134534Z namespaces/openshift-dns/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027184706Z namespaces/openshift-dns/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027264147Z namespaces/openshift-dns/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027331359Z namespaces/openshift-dns/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02736703Z namespaces/openshift-dns/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027471622Z namespaces/openshift-dns/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027551974Z namespaces/openshift-dns/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027764689Z namespaces/openshift-dns/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.027836541Z namespaces/openshift-dns/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028123828Z namespaces/openshift-dns/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028235191Z namespaces/openshift-dns/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028403895Z namespaces/openshift-dns/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028465406Z namespaces/openshift-dns/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028514188Z namespaces/openshift-dns/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028560549Z namespaces/openshift-dns/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02860608Z namespaces/openshift-dns/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028660271Z namespaces/openshift-dns/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028698372Z namespaces/openshift-dns/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028784394Z namespaces/openshift-dns/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028854196Z namespaces/openshift-dns/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028897117Z namespaces/openshift-dns/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028938208Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.028984389Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029038691Z namespaces/openshift-dns/monitoring.coreos.com/servicemonitors/dns-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029106012Z namespaces/openshift-dns/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029148803Z namespaces/openshift-dns/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029196664Z namespaces/openshift-dns/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029202895Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029235255Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.02942763Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029613354Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029782478Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.029993014Z namespaces/openshift-dns/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030222879Z namespaces/openshift-dns/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03023448Z namespaces/openshift-dns/pods/dns-default-6xhrz/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030299471Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns-default-6xhrz.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030381023Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030393703Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns/dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030399013Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns/dns/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030446155Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns/dns/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030553427Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030618189Z namespaces/openshift-dns/pods/dns-default-6xhrz/dns/dns/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030649779Z namespaces/openshift-dns/pods/dns-default-6xhrz/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03066384Z namespaces/openshift-dns/pods/dns-default-6xhrz/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03067027Z namespaces/openshift-dns/pods/dns-default-6xhrz/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030715151Z namespaces/openshift-dns/pods/dns-default-6xhrz/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030801713Z namespaces/openshift-dns/pods/dns-default-6xhrz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030892185Z namespaces/openshift-dns/pods/dns-default-6xhrz/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030927416Z namespaces/openshift-dns/pods/dns-default-fj468/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.030976347Z namespaces/openshift-dns/pods/dns-default-fj468/dns-default-fj468.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03150264Z namespaces/openshift-dns/pods/dns-default-fj468/dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031518021Z namespaces/openshift-dns/pods/dns-default-fj468/dns/dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031521751Z namespaces/openshift-dns/pods/dns-default-fj468/dns/dns/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031555492Z namespaces/openshift-dns/pods/dns-default-fj468/dns/dns/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031644244Z namespaces/openshift-dns/pods/dns-default-fj468/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031711635Z namespaces/openshift-dns/pods/dns-default-fj468/dns/dns/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031748536Z namespaces/openshift-dns/pods/dns-default-fj468/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031755596Z namespaces/openshift-dns/pods/dns-default-fj468/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031762457Z namespaces/openshift-dns/pods/dns-default-fj468/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031816268Z namespaces/openshift-dns/pods/dns-default-fj468/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03190571Z namespaces/openshift-dns/pods/dns-default-fj468/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031962682Z namespaces/openshift-dns/pods/dns-default-fj468/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.031999132Z namespaces/openshift-dns/pods/dns-default-snkfr/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032042973Z namespaces/openshift-dns/pods/dns-default-snkfr/dns-default-snkfr.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032141626Z namespaces/openshift-dns/pods/dns-default-snkfr/dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032152456Z namespaces/openshift-dns/pods/dns-default-snkfr/dns/dns/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032155986Z namespaces/openshift-dns/pods/dns-default-snkfr/dns/dns/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032207467Z namespaces/openshift-dns/pods/dns-default-snkfr/dns/dns/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032300299Z namespaces/openshift-dns/pods/dns-default-snkfr/dns/dns/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032357691Z namespaces/openshift-dns/pods/dns-default-snkfr/dns/dns/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032400162Z namespaces/openshift-dns/pods/dns-default-snkfr/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032409852Z namespaces/openshift-dns/pods/dns-default-snkfr/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032417433Z namespaces/openshift-dns/pods/dns-default-snkfr/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032454943Z namespaces/openshift-dns/pods/dns-default-snkfr/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032545085Z namespaces/openshift-dns/pods/dns-default-snkfr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032605887Z namespaces/openshift-dns/pods/dns-default-snkfr/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032653768Z namespaces/openshift-dns/pods/node-resolver-772qp/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032693069Z namespaces/openshift-dns/pods/node-resolver-772qp/node-resolver-772qp.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032782691Z namespaces/openshift-dns/pods/node-resolver-772qp/dns-node-resolver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032789631Z namespaces/openshift-dns/pods/node-resolver-772qp/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032793162Z namespaces/openshift-dns/pods/node-resolver-772qp/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032832863Z namespaces/openshift-dns/pods/node-resolver-772qp/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032919405Z namespaces/openshift-dns/pods/node-resolver-772qp/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.032979506Z namespaces/openshift-dns/pods/node-resolver-772qp/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033050928Z namespaces/openshift-dns/pods/node-resolver-llqwv/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033102699Z namespaces/openshift-dns/pods/node-resolver-llqwv/node-resolver-llqwv.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033173061Z namespaces/openshift-dns/pods/node-resolver-llqwv/dns-node-resolver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033179581Z namespaces/openshift-dns/pods/node-resolver-llqwv/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033182941Z namespaces/openshift-dns/pods/node-resolver-llqwv/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033209962Z namespaces/openshift-dns/pods/node-resolver-llqwv/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033293444Z namespaces/openshift-dns/pods/node-resolver-llqwv/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033359185Z namespaces/openshift-dns/pods/node-resolver-llqwv/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033390106Z namespaces/openshift-dns/pods/node-resolver-ncfhm/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033429747Z namespaces/openshift-dns/pods/node-resolver-ncfhm/node-resolver-ncfhm.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033500479Z namespaces/openshift-dns/pods/node-resolver-ncfhm/dns-node-resolver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033509929Z namespaces/openshift-dns/pods/node-resolver-ncfhm/dns-node-resolver/dns-node-resolver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033515599Z namespaces/openshift-dns/pods/node-resolver-ncfhm/dns-node-resolver/dns-node-resolver/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033528109Z namespaces/openshift-dns/pods/node-resolver-ncfhm/dns-node-resolver/dns-node-resolver/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033617862Z namespaces/openshift-dns/pods/node-resolver-ncfhm/dns-node-resolver/dns-node-resolver/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033678693Z namespaces/openshift-dns/pods/node-resolver-ncfhm/dns-node-resolver/dns-node-resolver/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033715564Z namespaces/openshift-dns/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033748595Z namespaces/openshift-dns/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033791996Z namespaces/openshift-dns/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033827557Z namespaces/openshift-dns/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.033917849Z namespaces/openshift-etcd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03396212Z namespaces/openshift-etcd/openshift-etcd.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034015991Z namespaces/openshift-etcd/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034081103Z namespaces/openshift-etcd/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034135494Z namespaces/openshift-etcd/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034173045Z namespaces/openshift-etcd/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034250157Z namespaces/openshift-etcd/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034325439Z namespaces/openshift-etcd/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034399641Z namespaces/openshift-etcd/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034441831Z namespaces/openshift-etcd/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034482703Z namespaces/openshift-etcd/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034532424Z namespaces/openshift-etcd/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034573045Z namespaces/openshift-etcd/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034651177Z namespaces/openshift-etcd/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034709048Z namespaces/openshift-etcd/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034757209Z namespaces/openshift-etcd/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034830521Z namespaces/openshift-etcd/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034908913Z namespaces/openshift-etcd/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.034950194Z namespaces/openshift-etcd/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035070727Z namespaces/openshift-etcd/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035149769Z namespaces/openshift-etcd/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03522595Z namespaces/openshift-etcd/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035299672Z namespaces/openshift-etcd/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035373274Z namespaces/openshift-etcd/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035492397Z namespaces/openshift-etcd/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035647241Z namespaces/openshift-etcd/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035693722Z namespaces/openshift-etcd/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035750033Z namespaces/openshift-etcd/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035785434Z namespaces/openshift-etcd/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035828975Z namespaces/openshift-etcd/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035881477Z namespaces/openshift-etcd/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035928618Z namespaces/openshift-etcd/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.035996569Z namespaces/openshift-etcd/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036111002Z namespaces/openshift-etcd/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036159253Z namespaces/openshift-etcd/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036197744Z namespaces/openshift-etcd/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036238075Z namespaces/openshift-etcd/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036277246Z namespaces/openshift-etcd/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036319977Z namespaces/openshift-etcd/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036346228Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036391629Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036530562Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.036705706Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03683526Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037041705Z namespaces/openshift-etcd/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037302931Z namespaces/openshift-etcd/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037336262Z namespaces/openshift-etcd/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037382613Z namespaces/openshift-etcd/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037420954Z namespaces/openshift-etcd/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037503116Z namespaces/openshift-host-network/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037549267Z namespaces/openshift-host-network/openshift-host-network.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037616468Z namespaces/openshift-host-network/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037653529Z namespaces/openshift-host-network/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037728901Z namespaces/openshift-host-network/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037764972Z namespaces/openshift-host-network/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037841564Z namespaces/openshift-host-network/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037912756Z namespaces/openshift-host-network/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.037990738Z namespaces/openshift-host-network/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038054239Z namespaces/openshift-host-network/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03810068Z namespaces/openshift-host-network/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038149872Z namespaces/openshift-host-network/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038187773Z namespaces/openshift-host-network/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038272184Z namespaces/openshift-host-network/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038323156Z namespaces/openshift-host-network/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038366497Z namespaces/openshift-host-network/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038441239Z namespaces/openshift-host-network/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03850402Z namespaces/openshift-host-network/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038548421Z namespaces/openshift-host-network/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038645273Z namespaces/openshift-host-network/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038716775Z namespaces/openshift-host-network/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038793057Z namespaces/openshift-host-network/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038881949Z namespaces/openshift-host-network/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.038955071Z namespaces/openshift-host-network/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039130095Z namespaces/openshift-host-network/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039229638Z namespaces/openshift-host-network/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039281629Z namespaces/openshift-host-network/core/resourcequotas/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.03932644Z namespaces/openshift-host-network/core/resourcequotas/host-network-namespace-quotas.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039379461Z namespaces/openshift-host-network/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039426623Z namespaces/openshift-host-network/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039472444Z namespaces/openshift-host-network/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039516995Z namespaces/openshift-host-network/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039567766Z namespaces/openshift-host-network/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039612157Z namespaces/openshift-host-network/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039690199Z namespaces/openshift-host-network/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039768721Z namespaces/openshift-host-network/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039808872Z namespaces/openshift-host-network/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039846933Z namespaces/openshift-host-network/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039888854Z namespaces/openshift-host-network/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039927785Z namespaces/openshift-host-network/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039979186Z namespaces/openshift-host-network/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.039991666Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.040056108Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.040222442Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.040402736Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.040530319Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.040734324Z namespaces/openshift-host-network/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.040935069Z namespaces/openshift-host-network/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04096964Z namespaces/openshift-host-network/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041012071Z namespaces/openshift-host-network/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041091223Z namespaces/openshift-host-network/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041163014Z namespaces/openshift-image-registry/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041210856Z namespaces/openshift-image-registry/openshift-image-registry.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041266117Z namespaces/openshift-image-registry/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041308178Z namespaces/openshift-image-registry/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041344159Z namespaces/openshift-image-registry/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04138967Z namespaces/openshift-image-registry/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041486373Z namespaces/openshift-image-registry/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041617746Z namespaces/openshift-image-registry/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04181254Z namespaces/openshift-image-registry/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041851291Z namespaces/openshift-image-registry/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041891912Z namespaces/openshift-image-registry/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041949234Z namespaces/openshift-image-registry/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.041991595Z namespaces/openshift-image-registry/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042124478Z namespaces/openshift-image-registry/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042163479Z namespaces/openshift-image-registry/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04220165Z namespaces/openshift-image-registry/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042279532Z namespaces/openshift-image-registry/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042320823Z namespaces/openshift-image-registry/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042333333Z namespaces/openshift-image-registry/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042380114Z namespaces/openshift-image-registry/coordination.k8s.io/leases/openshift-master-controllers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042453246Z namespaces/openshift-image-registry/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.042486067Z namespaces/openshift-image-registry/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.043473861Z namespaces/openshift-image-registry/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.043602914Z namespaces/openshift-image-registry/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.0438419Z namespaces/openshift-image-registry/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.043919261Z namespaces/openshift-image-registry/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044158637Z namespaces/openshift-image-registry/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04427477Z namespaces/openshift-image-registry/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044471865Z namespaces/openshift-image-registry/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044523876Z namespaces/openshift-image-registry/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044561977Z namespaces/openshift-image-registry/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044617278Z namespaces/openshift-image-registry/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04466884Z namespaces/openshift-image-registry/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044713581Z namespaces/openshift-image-registry/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044750142Z namespaces/openshift-image-registry/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044839704Z namespaces/openshift-image-registry/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044926356Z namespaces/openshift-image-registry/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.044984117Z namespaces/openshift-image-registry/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045020928Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045188512Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045256374Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-operator-alerts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045372197Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/image-registry-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04549594Z namespaces/openshift-image-registry/monitoring.coreos.com/prometheusrules/imagestreams-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045543561Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045591182Z namespaces/openshift-image-registry/monitoring.coreos.com/servicemonitors/image-registry.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045640793Z namespaces/openshift-image-registry/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045692235Z namespaces/openshift-image-registry/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045735616Z namespaces/openshift-image-registry/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045745006Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045805437Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.045957571Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046165316Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04630987Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046535355Z namespaces/openshift-image-registry/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04673875Z namespaces/openshift-image-registry/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.0467537Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046775441Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/image-registry-54dbbfbdcd-xq7ll.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046899584Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/registry/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046912724Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/registry/registry/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046916394Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/registry/registry/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.046952875Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/registry/registry/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049173329Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/registry/registry/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049250311Z namespaces/openshift-image-registry/pods/image-registry-54dbbfbdcd-xq7ll/registry/registry/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049268011Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049322093Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca-ffv5j.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049399614Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049411865Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca/node-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049416115Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca/node-ca/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049425515Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca/node-ca/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.04961214Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049678561Z namespaces/openshift-image-registry/pods/node-ca-ffv5j/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049716942Z namespaces/openshift-image-registry/pods/node-ca-jjncf/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049749933Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca-jjncf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049833035Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049845445Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca/node-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049852325Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca/node-ca/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.049885436Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca/node-ca/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050083601Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050148873Z namespaces/openshift-image-registry/pods/node-ca-jjncf/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050180764Z namespaces/openshift-image-registry/pods/node-ca-swv7c/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050224154Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca-swv7c.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050300996Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050316297Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca/node-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050320097Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca/node-ca/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050367288Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca/node-ca/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050541362Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca/node-ca/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050615264Z namespaces/openshift-image-registry/pods/node-ca-swv7c/node-ca/node-ca/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050627654Z namespaces/openshift-image-registry/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050682056Z namespaces/openshift-image-registry/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050741987Z namespaces/openshift-image-registry/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050773038Z namespaces/openshift-image-registry/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050821759Z namespaces/openshift-infra/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050830179Z namespaces/openshift-infra/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.050835569Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05087114Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051052925Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051230329Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051366952Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051558927Z namespaces/openshift-infra/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051772522Z namespaces/openshift-ingress-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051814713Z namespaces/openshift-ingress-canary/openshift-ingress-canary.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051887995Z namespaces/openshift-ingress-canary/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051927696Z namespaces/openshift-ingress-canary/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.051991647Z namespaces/openshift-ingress-canary/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052041749Z namespaces/openshift-ingress-canary/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052156201Z namespaces/openshift-ingress-canary/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052235073Z namespaces/openshift-ingress-canary/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052324865Z namespaces/openshift-ingress-canary/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052372077Z namespaces/openshift-ingress-canary/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052417988Z namespaces/openshift-ingress-canary/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052460519Z namespaces/openshift-ingress-canary/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05250226Z namespaces/openshift-ingress-canary/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052577231Z namespaces/openshift-ingress-canary/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052637933Z namespaces/openshift-ingress-canary/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052672614Z namespaces/openshift-ingress-canary/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052757906Z namespaces/openshift-ingress-canary/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052817608Z namespaces/openshift-ingress-canary/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052870609Z namespaces/openshift-ingress-canary/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.052989772Z namespaces/openshift-ingress-canary/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053100204Z namespaces/openshift-ingress-canary/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053241418Z namespaces/openshift-ingress-canary/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.0533186Z namespaces/openshift-ingress-canary/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053479344Z namespaces/openshift-ingress-canary/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053601607Z namespaces/openshift-ingress-canary/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05375839Z namespaces/openshift-ingress-canary/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053819732Z namespaces/openshift-ingress-canary/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053866153Z namespaces/openshift-ingress-canary/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053927484Z namespaces/openshift-ingress-canary/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.053973796Z namespaces/openshift-ingress-canary/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054021666Z namespaces/openshift-ingress-canary/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054085028Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.0541707Z namespaces/openshift-ingress-canary/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054243022Z namespaces/openshift-ingress-canary/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054296703Z namespaces/openshift-ingress-canary/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054338594Z namespaces/openshift-ingress-canary/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054395696Z namespaces/openshift-ingress-canary/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054434997Z namespaces/openshift-ingress-canary/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054495018Z namespaces/openshift-ingress-canary/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054503238Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054545659Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054695343Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.054893828Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055045602Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055258237Z namespaces/openshift-ingress-canary/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055642456Z namespaces/openshift-ingress-canary/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055650996Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055695647Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/ingress-canary-brdtq.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055773529Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/serve-healthcheck-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055781629Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05579116Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05582723Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055923843Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.055993265Z namespaces/openshift-ingress-canary/pods/ingress-canary-brdtq/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056065046Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056099797Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/ingress-canary-r64tx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056176719Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/serve-healthcheck-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056187269Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056196769Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05623301Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056327582Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056396954Z namespaces/openshift-ingress-canary/pods/ingress-canary-r64tx/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056447996Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056481716Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/ingress-canary-s6d8g.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056564478Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/serve-healthcheck-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056572099Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/serve-healthcheck-canary/serve-healthcheck-canary/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056577949Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/serve-healthcheck-canary/serve-healthcheck-canary/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056616729Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/serve-healthcheck-canary/serve-healthcheck-canary/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056715892Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056778433Z namespaces/openshift-ingress-canary/pods/ingress-canary-s6d8g/serve-healthcheck-canary/serve-healthcheck-canary/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056819594Z namespaces/openshift-ingress-canary/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056856226Z namespaces/openshift-ingress-canary/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056911807Z namespaces/openshift-ingress-canary/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.056950228Z namespaces/openshift-ingress-canary/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057070641Z namespaces/openshift-ingress-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057120102Z namespaces/openshift-ingress-operator/openshift-ingress-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057187443Z namespaces/openshift-ingress-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057228055Z namespaces/openshift-ingress-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057285766Z namespaces/openshift-ingress-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057329577Z namespaces/openshift-ingress-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057403869Z namespaces/openshift-ingress-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057482601Z namespaces/openshift-ingress-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057560352Z namespaces/openshift-ingress-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057612754Z namespaces/openshift-ingress-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057656255Z namespaces/openshift-ingress-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057700416Z namespaces/openshift-ingress-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057741727Z namespaces/openshift-ingress-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057819919Z namespaces/openshift-ingress-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05787394Z namespaces/openshift-ingress-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057918881Z namespaces/openshift-ingress-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.057994763Z namespaces/openshift-ingress-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.058079505Z namespaces/openshift-ingress-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.058122336Z namespaces/openshift-ingress-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059042108Z namespaces/openshift-ingress-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059142991Z namespaces/openshift-ingress-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059260674Z namespaces/openshift-ingress-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059338886Z namespaces/openshift-ingress-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059421468Z namespaces/openshift-ingress-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059541951Z namespaces/openshift-ingress-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059728335Z namespaces/openshift-ingress-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059794557Z namespaces/openshift-ingress-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059839308Z namespaces/openshift-ingress-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.05991984Z namespaces/openshift-ingress-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.059962911Z namespaces/openshift-ingress-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060012912Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060020482Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060084364Z namespaces/openshift-ingress-operator/ingress.operator.openshift.io/dnsrecords/default-wildcard.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060153915Z namespaces/openshift-ingress-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060202116Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060285458Z namespaces/openshift-ingress-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060387211Z namespaces/openshift-ingress-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060446922Z namespaces/openshift-ingress-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060493004Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060552795Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060605286Z namespaces/openshift-ingress-operator/monitoring.coreos.com/prometheusrules/ingress-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060672228Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060718069Z namespaces/openshift-ingress-operator/monitoring.coreos.com/servicemonitors/ingress-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06076886Z namespaces/openshift-ingress-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060818302Z namespaces/openshift-ingress-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060857072Z namespaces/openshift-ingress-operator/operator.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060867933Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060920994Z namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.060987956Z namespaces/openshift-ingress-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.061002636Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.061088248Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.061244062Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.061423126Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06159455Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.061800365Z namespaces/openshift-ingress-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06200219Z namespaces/openshift-ingress-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062047421Z namespaces/openshift-ingress-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062114583Z namespaces/openshift-ingress-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062158204Z namespaces/openshift-ingress-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062252066Z namespaces/openshift-ingress/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062295477Z namespaces/openshift-ingress/openshift-ingress.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062350399Z namespaces/openshift-ingress/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06239205Z namespaces/openshift-ingress/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062455281Z namespaces/openshift-ingress/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062493832Z namespaces/openshift-ingress/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.062574144Z namespaces/openshift-ingress/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06279325Z namespaces/openshift-ingress/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063011235Z namespaces/openshift-ingress/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063077306Z namespaces/openshift-ingress/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063122018Z namespaces/openshift-ingress/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063193869Z namespaces/openshift-ingress/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06323195Z namespaces/openshift-ingress/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063308672Z namespaces/openshift-ingress/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063362193Z namespaces/openshift-ingress/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063407714Z namespaces/openshift-ingress/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063485136Z namespaces/openshift-ingress/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063728552Z namespaces/openshift-ingress/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063740073Z namespaces/openshift-ingress/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063791524Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-ca-openshift-gateway.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063877116Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-deployment-openshift-gateway.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.063959598Z namespaces/openshift-ingress/coordination.k8s.io/leases/istio-gateway-status-leader-openshift-gateway.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.064087611Z namespaces/openshift-ingress/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.064116752Z namespaces/openshift-ingress/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.064527381Z namespaces/openshift-ingress/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.064632804Z namespaces/openshift-ingress/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.064837599Z namespaces/openshift-ingress/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.064917391Z namespaces/openshift-ingress/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065185628Z namespaces/openshift-ingress/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06530317Z namespaces/openshift-ingress/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065484375Z namespaces/openshift-ingress/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065563517Z namespaces/openshift-ingress/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065608278Z namespaces/openshift-ingress/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06568993Z namespaces/openshift-ingress/gateway.networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06569846Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065751991Z namespaces/openshift-ingress/gateway.networking.k8s.io/gateways/openshift-ai-inference.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065833433Z namespaces/openshift-ingress/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065875764Z namespaces/openshift-ingress/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065928015Z namespaces/openshift-ingress/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.065970677Z namespaces/openshift-ingress/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066074499Z namespaces/openshift-ingress/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066151251Z namespaces/openshift-ingress/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066214322Z namespaces/openshift-ingress/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066256403Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066312135Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066363766Z namespaces/openshift-ingress/monitoring.coreos.com/servicemonitors/router-default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066401617Z namespaces/openshift-ingress/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066450228Z namespaces/openshift-ingress/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066494329Z namespaces/openshift-ingress/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.0665162Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066566731Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.066716985Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06693788Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067101304Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067318269Z namespaces/openshift-ingress/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067516864Z namespaces/openshift-ingress/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067528784Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067562885Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/istiod-openshift-gateway-94bb8fbfd-kqnw7.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067678188Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067685668Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.067691828Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.06773667Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.09744006Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097504622Z namespaces/openshift-ingress/pods/istiod-openshift-gateway-94bb8fbfd-kqnw7/discovery/discovery/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097561203Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097594254Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/openshift-ai-inference-openshift-default-655f7c58b4-57g8x.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097731147Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097738997Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097744087Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.097785308Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.098396013Z namespaces/openshift-ingress/pods/openshift-ai-inference-openshift-default-655f7c58b4-57g8x/istio-proxy/istio-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099539551Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099561851Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router-default-bcd57767c-rwpfk.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099673594Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099682654Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099686734Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099728516Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099847329Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.09991468Z namespaces/openshift-ingress/pods/router-default-bcd57767c-rwpfk/router/router/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.099962881Z namespaces/openshift-ingress/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100005732Z namespaces/openshift-ingress/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100077054Z namespaces/openshift-ingress/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100117825Z namespaces/openshift-ingress/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100208607Z namespaces/openshift-insights/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100264809Z namespaces/openshift-insights/openshift-insights.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10032636Z namespaces/openshift-insights/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100377491Z namespaces/openshift-insights/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100432353Z namespaces/openshift-insights/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100478534Z namespaces/openshift-insights/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100592136Z namespaces/openshift-insights/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100685599Z namespaces/openshift-insights/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100789731Z namespaces/openshift-insights/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100827392Z namespaces/openshift-insights/apps/deployments/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100875423Z namespaces/openshift-insights/apps/deployments/insights-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100937665Z namespaces/openshift-insights/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.100989466Z namespaces/openshift-insights/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101061708Z namespaces/openshift-insights/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101100439Z namespaces/openshift-insights/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101182971Z namespaces/openshift-insights/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101234842Z namespaces/openshift-insights/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101278373Z namespaces/openshift-insights/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101354255Z namespaces/openshift-insights/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101420077Z namespaces/openshift-insights/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.101467338Z namespaces/openshift-insights/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10236441Z namespaces/openshift-insights/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.102445052Z namespaces/openshift-insights/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.102716448Z namespaces/openshift-insights/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.102864382Z namespaces/openshift-insights/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10318579Z namespaces/openshift-insights/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103268932Z namespaces/openshift-insights/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103425595Z namespaces/openshift-insights/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103486427Z namespaces/openshift-insights/core/configmaps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103533748Z namespaces/openshift-insights/core/configmaps/service-ca-bundle.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103588589Z namespaces/openshift-insights/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10363401Z namespaces/openshift-insights/core/serviceaccounts/gather.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103718752Z namespaces/openshift-insights/core/serviceaccounts/operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103769534Z namespaces/openshift-insights/core/services/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103820915Z namespaces/openshift-insights/core/services/metrics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103876626Z namespaces/openshift-insights/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103922577Z namespaces/openshift-insights/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.103978319Z namespaces/openshift-insights/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10402333Z namespaces/openshift-insights/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104091972Z namespaces/openshift-insights/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104128513Z namespaces/openshift-insights/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104212844Z namespaces/openshift-insights/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104291326Z namespaces/openshift-insights/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104343118Z namespaces/openshift-insights/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104392279Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1044467Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104497911Z namespaces/openshift-insights/monitoring.coreos.com/prometheusrules/insights-prometheus-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104557963Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104598564Z namespaces/openshift-insights/monitoring.coreos.com/servicemonitors/insights-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104662075Z namespaces/openshift-insights/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104704586Z namespaces/openshift-insights/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104753838Z namespaces/openshift-insights/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104766928Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.104810409Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105000454Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10525429Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105405284Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105607668Z namespaces/openshift-insights/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105827244Z namespaces/openshift-insights/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105834104Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105873635Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/insights-operator-5f67b55555-zs75b.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105961807Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/insights-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105968907Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/insights-operator/insights-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.105976657Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/insights-operator/insights-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.106039339Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/insights-operator/insights-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.106451719Z namespaces/openshift-insights/pods/insights-operator-5f67b55555-zs75b/insights-operator/insights-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.106739116Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.106780177Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/insights-runtime-extractor-45r8m.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.106886929Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10689837Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/exporter/exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1069025Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/exporter/exporter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10692894Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/exporter/exporter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107020623Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107121575Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/exporter/exporter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107138726Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/extractor/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107144575Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/extractor/extractor/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107147796Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/extractor/extractor/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107195207Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/extractor/extractor/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107289029Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107355651Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/extractor/extractor/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107392672Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107399352Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107420662Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107469454Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107555605Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107618567Z namespaces/openshift-insights/pods/insights-runtime-extractor-45r8m/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107671919Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107711279Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/insights-runtime-extractor-sk28n.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107794461Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107801471Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/exporter/exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107810612Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/exporter/exporter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107860933Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/exporter/exporter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.107947655Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108013617Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/exporter/exporter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108074328Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/extractor/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108088838Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/extractor/extractor/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108097839Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/extractor/extractor/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10813283Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/extractor/extractor/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108228692Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108295003Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/extractor/extractor/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108330014Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108340115Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108343385Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108376925Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108460567Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108528519Z namespaces/openshift-insights/pods/insights-runtime-extractor-sk28n/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108710964Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108754785Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/insights-runtime-extractor-wpqxs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108837347Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108844427Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/exporter/exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108850217Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/exporter/exporter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.108889788Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/exporter/exporter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10897768Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/exporter/exporter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109086573Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/exporter/exporter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109106703Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/extractor/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109113613Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/extractor/extractor/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109117374Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/extractor/extractor/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109163944Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/extractor/extractor/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109264137Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/extractor/extractor/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109329219Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/extractor/extractor/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109366689Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10937343Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10937693Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109416201Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109500303Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109571174Z namespaces/openshift-insights/pods/insights-runtime-extractor-wpqxs/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109605755Z namespaces/openshift-insights/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109647256Z namespaces/openshift-insights/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109695257Z namespaces/openshift-insights/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109735818Z namespaces/openshift-insights/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10978525Z namespaces/openshift-keda/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10979122Z namespaces/openshift-keda/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.10979674Z namespaces/openshift-keda/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109834531Z namespaces/openshift-keda/coordination.k8s.io/leases/olm-operator.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109928373Z namespaces/openshift-keda/coordination.k8s.io/leases/operator.keda.sh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109971784Z namespaces/openshift-keda/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.109979324Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110018925Z namespaces/openshift-keda/monitoring.coreos.com/podmonitors/keda-olm-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110093907Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110136688Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-admission-webhooks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11022213Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-metrics-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110304302Z namespaces/openshift-keda/monitoring.coreos.com/servicemonitors/keda-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110355733Z namespaces/openshift-keda/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110367474Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110419255Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110558118Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110731483Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.110968268Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111228305Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111377898Z namespaces/openshift-keda/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111548542Z namespaces/openshift-keda/operators.coreos.com/installplans/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111625674Z namespaces/openshift-keda/operators.coreos.com/installplans/install-jvqjf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111724987Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111795519Z namespaces/openshift-keda/operators.coreos.com/operatorconditions/custom-metrics-autoscaler.v2.19.0-1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11184441Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111890321Z namespaces/openshift-keda/operators.coreos.com/operatorgroups/openshift-keda.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111941672Z namespaces/openshift-keda/operators.coreos.com/subscriptions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.111994613Z namespaces/openshift-keda/operators.coreos.com/subscriptions/openshift-custom-metrics-autoscaler-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112100786Z namespaces/openshift-kube-apiserver-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112147247Z namespaces/openshift-kube-apiserver-operator/openshift-kube-apiserver-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112202258Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112236689Z namespaces/openshift-kube-apiserver-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112309341Z namespaces/openshift-kube-apiserver-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112348422Z namespaces/openshift-kube-apiserver-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112438974Z namespaces/openshift-kube-apiserver-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112503026Z namespaces/openshift-kube-apiserver-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112581338Z namespaces/openshift-kube-apiserver-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112633319Z namespaces/openshift-kube-apiserver-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11267078Z namespaces/openshift-kube-apiserver-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112724891Z namespaces/openshift-kube-apiserver-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112761752Z namespaces/openshift-kube-apiserver-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112840034Z namespaces/openshift-kube-apiserver-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112903695Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.112938856Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113047459Z namespaces/openshift-kube-apiserver-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11311211Z namespaces/openshift-kube-apiserver-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113157632Z namespaces/openshift-kube-apiserver-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113261494Z namespaces/openshift-kube-apiserver-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113332316Z namespaces/openshift-kube-apiserver-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113411638Z namespaces/openshift-kube-apiserver-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11349092Z namespaces/openshift-kube-apiserver-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113565981Z namespaces/openshift-kube-apiserver-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113690794Z namespaces/openshift-kube-apiserver-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113859189Z namespaces/openshift-kube-apiserver-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11391475Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.113962281Z namespaces/openshift-kube-apiserver-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114013322Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114085854Z namespaces/openshift-kube-apiserver-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114140645Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114188486Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114265288Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11434308Z namespaces/openshift-kube-apiserver-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114386081Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114427892Z namespaces/openshift-kube-apiserver-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114477304Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114512964Z namespaces/openshift-kube-apiserver-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114559575Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114577036Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114609667Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114769851Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.114968005Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115126229Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115311854Z namespaces/openshift-kube-apiserver-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115497288Z namespaces/openshift-kube-apiserver-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115538499Z namespaces/openshift-kube-apiserver-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115603341Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115636532Z namespaces/openshift-kube-apiserver-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115735874Z namespaces/openshift-kube-apiserver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115777645Z namespaces/openshift-kube-apiserver/openshift-kube-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115836686Z namespaces/openshift-kube-apiserver/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115863687Z namespaces/openshift-kube-apiserver/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.115939279Z namespaces/openshift-kube-apiserver/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11597253Z namespaces/openshift-kube-apiserver/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116077122Z namespaces/openshift-kube-apiserver/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116152594Z namespaces/openshift-kube-apiserver/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116229266Z namespaces/openshift-kube-apiserver/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116278157Z namespaces/openshift-kube-apiserver/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116315108Z namespaces/openshift-kube-apiserver/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11637243Z namespaces/openshift-kube-apiserver/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11640875Z namespaces/openshift-kube-apiserver/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116490072Z namespaces/openshift-kube-apiserver/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116543944Z namespaces/openshift-kube-apiserver/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116589155Z namespaces/openshift-kube-apiserver/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116667987Z namespaces/openshift-kube-apiserver/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116735118Z namespaces/openshift-kube-apiserver/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116779409Z namespaces/openshift-kube-apiserver/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116871531Z namespaces/openshift-kube-apiserver/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.116945053Z namespaces/openshift-kube-apiserver/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117042926Z namespaces/openshift-kube-apiserver/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117131118Z namespaces/openshift-kube-apiserver/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117202119Z namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117326673Z namespaces/openshift-kube-apiserver/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117486597Z namespaces/openshift-kube-apiserver/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117528867Z namespaces/openshift-kube-apiserver/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117578189Z namespaces/openshift-kube-apiserver/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11761465Z namespaces/openshift-kube-apiserver/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117663471Z namespaces/openshift-kube-apiserver/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117701812Z namespaces/openshift-kube-apiserver/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117746493Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117822625Z namespaces/openshift-kube-apiserver/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117897047Z namespaces/openshift-kube-apiserver/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117949418Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.117992669Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118086731Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118130352Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/api-usage.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118224955Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/prometheusrules/podsecurity.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118269956Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118314607Z namespaces/openshift-kube-apiserver/monitoring.coreos.com/servicemonitors/openshift-kube-apiserver.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118389119Z namespaces/openshift-kube-apiserver/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118426919Z namespaces/openshift-kube-apiserver/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11847721Z namespaces/openshift-kube-apiserver/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118486431Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118524512Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118670155Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11886157Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.118997193Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119217379Z namespaces/openshift-kube-apiserver/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119404853Z namespaces/openshift-kube-apiserver/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119443304Z namespaces/openshift-kube-apiserver/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119494965Z namespaces/openshift-kube-apiserver/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119523406Z namespaces/openshift-kube-apiserver/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119589608Z namespaces/openshift-kube-controller-manager-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119628169Z namespaces/openshift-kube-controller-manager-operator/openshift-kube-controller-manager-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.11968195Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119724571Z namespaces/openshift-kube-controller-manager-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119780512Z namespaces/openshift-kube-controller-manager-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119821403Z namespaces/openshift-kube-controller-manager-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119898665Z namespaces/openshift-kube-controller-manager-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.119978187Z namespaces/openshift-kube-controller-manager-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1200784Z namespaces/openshift-kube-controller-manager-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120126471Z namespaces/openshift-kube-controller-manager-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120162631Z namespaces/openshift-kube-controller-manager-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120216193Z namespaces/openshift-kube-controller-manager-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120261194Z namespaces/openshift-kube-controller-manager-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120332866Z namespaces/openshift-kube-controller-manager-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120378857Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120419518Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12049706Z namespaces/openshift-kube-controller-manager-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120549521Z namespaces/openshift-kube-controller-manager-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120594122Z namespaces/openshift-kube-controller-manager-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120693614Z namespaces/openshift-kube-controller-manager-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120771746Z namespaces/openshift-kube-controller-manager-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.120852248Z namespaces/openshift-kube-controller-manager-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12092633Z namespaces/openshift-kube-controller-manager-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121007772Z namespaces/openshift-kube-controller-manager-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121153686Z namespaces/openshift-kube-controller-manager-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12133429Z namespaces/openshift-kube-controller-manager-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121391371Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121437642Z namespaces/openshift-kube-controller-manager-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121479934Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121525864Z namespaces/openshift-kube-controller-manager-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121558825Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121609487Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121683758Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12175923Z namespaces/openshift-kube-controller-manager-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121805881Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121846872Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121897333Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121941745Z namespaces/openshift-kube-controller-manager-operator/monitoring.coreos.com/servicemonitors/kube-controller-manager-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.121992986Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122055177Z namespaces/openshift-kube-controller-manager-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122102069Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122113389Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12214876Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122296763Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122472878Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122604231Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.122805386Z namespaces/openshift-kube-controller-manager-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1230085Z namespaces/openshift-kube-controller-manager-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123066172Z namespaces/openshift-kube-controller-manager-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123126914Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123166255Z namespaces/openshift-kube-controller-manager-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123243676Z namespaces/openshift-kube-controller-manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123289027Z namespaces/openshift-kube-controller-manager/openshift-kube-controller-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123342559Z namespaces/openshift-kube-controller-manager/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12338393Z namespaces/openshift-kube-controller-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123440581Z namespaces/openshift-kube-controller-manager/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123471702Z namespaces/openshift-kube-controller-manager/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123561554Z namespaces/openshift-kube-controller-manager/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123634476Z namespaces/openshift-kube-controller-manager/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123713028Z namespaces/openshift-kube-controller-manager/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123764819Z namespaces/openshift-kube-controller-manager/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12380189Z namespaces/openshift-kube-controller-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123844071Z namespaces/openshift-kube-controller-manager/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123885822Z namespaces/openshift-kube-controller-manager/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.123965974Z namespaces/openshift-kube-controller-manager/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124008975Z namespaces/openshift-kube-controller-manager/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124085307Z namespaces/openshift-kube-controller-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124156618Z namespaces/openshift-kube-controller-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124198149Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124205209Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12424205Z namespaces/openshift-kube-controller-manager/coordination.k8s.io/leases/cluster-policy-controller-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124307012Z namespaces/openshift-kube-controller-manager/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124351283Z namespaces/openshift-kube-controller-manager/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124447466Z namespaces/openshift-kube-controller-manager/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124523537Z namespaces/openshift-kube-controller-manager/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124814034Z namespaces/openshift-kube-controller-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124889576Z namespaces/openshift-kube-controller-manager/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.124966238Z namespaces/openshift-kube-controller-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125202444Z namespaces/openshift-kube-controller-manager/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125380318Z namespaces/openshift-kube-controller-manager/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125419179Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12546744Z namespaces/openshift-kube-controller-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125511821Z namespaces/openshift-kube-controller-manager/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125555212Z namespaces/openshift-kube-controller-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125611174Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125654365Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125731487Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125819439Z namespaces/openshift-kube-controller-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12585338Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12589593Z namespaces/openshift-kube-controller-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125939102Z namespaces/openshift-kube-controller-manager/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.125980253Z namespaces/openshift-kube-controller-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126042934Z namespaces/openshift-kube-controller-manager/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126059774Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126093475Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126250579Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126423183Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126559337Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126758551Z namespaces/openshift-kube-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126953606Z namespaces/openshift-kube-controller-manager/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.126991547Z namespaces/openshift-kube-controller-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127054409Z namespaces/openshift-kube-controller-manager/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12710375Z namespaces/openshift-kube-controller-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127164161Z namespaces/openshift-kube-scheduler-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127208682Z namespaces/openshift-kube-scheduler-operator/openshift-kube-scheduler-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127266354Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127306445Z namespaces/openshift-kube-scheduler-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127365156Z namespaces/openshift-kube-scheduler-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127401777Z namespaces/openshift-kube-scheduler-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127486309Z namespaces/openshift-kube-scheduler-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127562221Z namespaces/openshift-kube-scheduler-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127634983Z namespaces/openshift-kube-scheduler-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127680734Z namespaces/openshift-kube-scheduler-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127713875Z namespaces/openshift-kube-scheduler-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127760556Z namespaces/openshift-kube-scheduler-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127800827Z namespaces/openshift-kube-scheduler-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127878549Z namespaces/openshift-kube-scheduler-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12791876Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.127963111Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128063263Z namespaces/openshift-kube-scheduler-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128142865Z namespaces/openshift-kube-scheduler-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128183716Z namespaces/openshift-kube-scheduler-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128286939Z namespaces/openshift-kube-scheduler-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12836161Z namespaces/openshift-kube-scheduler-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128437432Z namespaces/openshift-kube-scheduler-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128514674Z namespaces/openshift-kube-scheduler-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128597316Z namespaces/openshift-kube-scheduler-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128726379Z namespaces/openshift-kube-scheduler-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128910094Z namespaces/openshift-kube-scheduler-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128950375Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.128995976Z namespaces/openshift-kube-scheduler-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129068648Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129105968Z namespaces/openshift-kube-scheduler-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12917374Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129207631Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129292603Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129367175Z namespaces/openshift-kube-scheduler-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129410216Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129450237Z namespaces/openshift-kube-scheduler-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129494098Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129535069Z namespaces/openshift-kube-scheduler-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12957742Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.12959034Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129639392Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129788835Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.129961279Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130118443Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130333688Z namespaces/openshift-kube-scheduler-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130528033Z namespaces/openshift-kube-scheduler-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130564044Z namespaces/openshift-kube-scheduler-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130617445Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130657536Z namespaces/openshift-kube-scheduler-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130734578Z namespaces/openshift-kube-scheduler/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130773159Z namespaces/openshift-kube-scheduler/openshift-kube-scheduler.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13082329Z namespaces/openshift-kube-scheduler/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130869651Z namespaces/openshift-kube-scheduler/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130920423Z namespaces/openshift-kube-scheduler/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.130950373Z namespaces/openshift-kube-scheduler/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131049116Z namespaces/openshift-kube-scheduler/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131132078Z namespaces/openshift-kube-scheduler/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13120767Z namespaces/openshift-kube-scheduler/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131259561Z namespaces/openshift-kube-scheduler/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131292672Z namespaces/openshift-kube-scheduler/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131354153Z namespaces/openshift-kube-scheduler/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131398234Z namespaces/openshift-kube-scheduler/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131479326Z namespaces/openshift-kube-scheduler/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131521137Z namespaces/openshift-kube-scheduler/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131569108Z namespaces/openshift-kube-scheduler/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13163896Z namespaces/openshift-kube-scheduler/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131703602Z namespaces/openshift-kube-scheduler/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131751503Z namespaces/openshift-kube-scheduler/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131848285Z namespaces/openshift-kube-scheduler/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.131923707Z namespaces/openshift-kube-scheduler/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132005109Z namespaces/openshift-kube-scheduler/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132109661Z namespaces/openshift-kube-scheduler/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132185743Z namespaces/openshift-kube-scheduler/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132322057Z namespaces/openshift-kube-scheduler/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132504231Z namespaces/openshift-kube-scheduler/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132566272Z namespaces/openshift-kube-scheduler/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132614184Z namespaces/openshift-kube-scheduler/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132653154Z namespaces/openshift-kube-scheduler/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132692855Z namespaces/openshift-kube-scheduler/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132741297Z namespaces/openshift-kube-scheduler/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132785258Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13285879Z namespaces/openshift-kube-scheduler/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132940152Z namespaces/openshift-kube-scheduler/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.132983722Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133043794Z namespaces/openshift-kube-scheduler/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133145877Z namespaces/openshift-kube-scheduler/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133187107Z namespaces/openshift-kube-scheduler/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133236009Z namespaces/openshift-kube-scheduler/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133249069Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13327725Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133428113Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133607588Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133742181Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.133945436Z namespaces/openshift-kube-scheduler/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134146601Z namespaces/openshift-kube-scheduler/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134171381Z namespaces/openshift-kube-scheduler/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134227053Z namespaces/openshift-kube-scheduler/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134268944Z namespaces/openshift-kube-scheduler/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134333585Z namespaces/openshift-kube-storage-version-migrator-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134381236Z namespaces/openshift-kube-storage-version-migrator-operator/openshift-kube-storage-version-migrator-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134434588Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134463778Z namespaces/openshift-kube-storage-version-migrator-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13451865Z namespaces/openshift-kube-storage-version-migrator-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134559691Z namespaces/openshift-kube-storage-version-migrator-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134642093Z namespaces/openshift-kube-storage-version-migrator-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134750185Z namespaces/openshift-kube-storage-version-migrator-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134866338Z namespaces/openshift-kube-storage-version-migrator-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1349199Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.134957501Z namespaces/openshift-kube-storage-version-migrator-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135009172Z namespaces/openshift-kube-storage-version-migrator-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135076333Z namespaces/openshift-kube-storage-version-migrator-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135153915Z namespaces/openshift-kube-storage-version-migrator-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135202576Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135242987Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135322939Z namespaces/openshift-kube-storage-version-migrator-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13536202Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135368131Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135415152Z namespaces/openshift-kube-storage-version-migrator-operator/coordination.k8s.io/leases/openshift-kube-storage-version-migrator-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135469343Z namespaces/openshift-kube-storage-version-migrator-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135514254Z namespaces/openshift-kube-storage-version-migrator-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135621377Z namespaces/openshift-kube-storage-version-migrator-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135700048Z namespaces/openshift-kube-storage-version-migrator-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135858372Z namespaces/openshift-kube-storage-version-migrator-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.135935944Z namespaces/openshift-kube-storage-version-migrator-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136071057Z namespaces/openshift-kube-storage-version-migrator-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13619371Z namespaces/openshift-kube-storage-version-migrator-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136365364Z namespaces/openshift-kube-storage-version-migrator-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136419406Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136463047Z namespaces/openshift-kube-storage-version-migrator-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136507558Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136547589Z namespaces/openshift-kube-storage-version-migrator-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13659483Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136645201Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136710183Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136789455Z namespaces/openshift-kube-storage-version-migrator-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136838556Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136874167Z namespaces/openshift-kube-storage-version-migrator-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136920548Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.136962679Z namespaces/openshift-kube-storage-version-migrator-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13701284Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137018791Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137084742Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137227856Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1374052Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137539973Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137721188Z namespaces/openshift-kube-storage-version-migrator-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137899642Z namespaces/openshift-kube-storage-version-migrator-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137908012Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.137955263Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/kube-storage-version-migrator-operator-79964ff6f9-6j4dv.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138112387Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/kube-storage-version-migrator-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138122507Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138128397Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138186289Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138307102Z namespaces/openshift-kube-storage-version-migrator-operator/pods/kube-storage-version-migrator-operator-79964ff6f9-6j4dv/kube-storage-version-migrator-operator/kube-storage-version-migrator-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138451775Z namespaces/openshift-kube-storage-version-migrator-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138473926Z namespaces/openshift-kube-storage-version-migrator-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138527397Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138564758Z namespaces/openshift-kube-storage-version-migrator-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138618209Z namespaces/openshift-kube-storage-version-migrator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13866404Z namespaces/openshift-kube-storage-version-migrator/openshift-kube-storage-version-migrator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138737892Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138779653Z namespaces/openshift-kube-storage-version-migrator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138832444Z namespaces/openshift-kube-storage-version-migrator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138883836Z namespaces/openshift-kube-storage-version-migrator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.138963868Z namespaces/openshift-kube-storage-version-migrator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139094241Z namespaces/openshift-kube-storage-version-migrator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139185593Z namespaces/openshift-kube-storage-version-migrator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139229404Z namespaces/openshift-kube-storage-version-migrator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139275795Z namespaces/openshift-kube-storage-version-migrator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139318856Z namespaces/openshift-kube-storage-version-migrator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139359977Z namespaces/openshift-kube-storage-version-migrator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139439509Z namespaces/openshift-kube-storage-version-migrator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13948838Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139531641Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139614783Z namespaces/openshift-kube-storage-version-migrator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139656604Z namespaces/openshift-kube-storage-version-migrator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139696085Z namespaces/openshift-kube-storage-version-migrator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139791998Z namespaces/openshift-kube-storage-version-migrator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.13986494Z namespaces/openshift-kube-storage-version-migrator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.139980002Z namespaces/openshift-kube-storage-version-migrator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140083805Z namespaces/openshift-kube-storage-version-migrator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140197228Z namespaces/openshift-kube-storage-version-migrator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14032228Z namespaces/openshift-kube-storage-version-migrator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140493695Z namespaces/openshift-kube-storage-version-migrator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140543326Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140578237Z namespaces/openshift-kube-storage-version-migrator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140631808Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140673929Z namespaces/openshift-kube-storage-version-migrator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14072144Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140766711Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140839873Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140915425Z namespaces/openshift-kube-storage-version-migrator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.140958336Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141000767Z namespaces/openshift-kube-storage-version-migrator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14112828Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141167291Z namespaces/openshift-kube-storage-version-migrator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141217982Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141224442Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141262894Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141410687Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141582151Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141716454Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.141907359Z namespaces/openshift-kube-storage-version-migrator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142114014Z namespaces/openshift-kube-storage-version-migrator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142128384Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142147395Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator-786f9fbc5-2hghb.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142231127Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/graceful-termination/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142237747Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/graceful-termination/graceful-termination/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142243227Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/graceful-termination/graceful-termination/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142290859Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/graceful-termination/graceful-termination/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142380391Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/graceful-termination/graceful-termination/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142442582Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/graceful-termination/graceful-termination/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142474003Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142480393Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator/migrator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142483733Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator/migrator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142514614Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator/migrator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142608966Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator/migrator/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142677478Z namespaces/openshift-kube-storage-version-migrator/pods/migrator-786f9fbc5-2hghb/migrator/migrator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142708178Z namespaces/openshift-kube-storage-version-migrator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14274539Z namespaces/openshift-kube-storage-version-migrator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14279348Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142834281Z namespaces/openshift-kube-storage-version-migrator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142907193Z namespaces/openshift-lws-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.142948874Z namespaces/openshift-lws-operator/openshift-lws-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143007316Z namespaces/openshift-lws-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143059687Z namespaces/openshift-lws-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143112008Z namespaces/openshift-lws-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143155129Z namespaces/openshift-lws-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143236411Z namespaces/openshift-lws-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143371345Z namespaces/openshift-lws-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143489957Z namespaces/openshift-lws-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143541589Z namespaces/openshift-lws-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14358607Z namespaces/openshift-lws-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143628081Z namespaces/openshift-lws-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143668352Z namespaces/openshift-lws-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143744354Z namespaces/openshift-lws-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143784325Z namespaces/openshift-lws-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143825396Z namespaces/openshift-lws-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143900938Z namespaces/openshift-lws-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143944259Z namespaces/openshift-lws-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.143950109Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14398803Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/b8b2488c.x-k8s.io.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144093782Z namespaces/openshift-lws-operator/coordination.k8s.io/leases/openshift-lws-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144150073Z namespaces/openshift-lws-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144193845Z namespaces/openshift-lws-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144294827Z namespaces/openshift-lws-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144378369Z namespaces/openshift-lws-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144685037Z namespaces/openshift-lws-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144761508Z namespaces/openshift-lws-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.144931883Z namespaces/openshift-lws-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145067716Z namespaces/openshift-lws-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145268971Z namespaces/openshift-lws-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145348073Z namespaces/openshift-lws-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145387423Z namespaces/openshift-lws-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145450275Z namespaces/openshift-lws-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145486736Z namespaces/openshift-lws-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145534827Z namespaces/openshift-lws-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145575318Z namespaces/openshift-lws-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14565379Z namespaces/openshift-lws-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145729272Z namespaces/openshift-lws-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145777133Z namespaces/openshift-lws-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145818234Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145876906Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145917487Z namespaces/openshift-lws-operator/monitoring.coreos.com/servicemonitors/lws-controller-manager-metrics-monitor.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.145968068Z namespaces/openshift-lws-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146010279Z namespaces/openshift-lws-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146082031Z namespaces/openshift-lws-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146095131Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146131792Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146282185Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14646679Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146602543Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.146829298Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147012343Z namespaces/openshift-lws-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147208298Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147252349Z namespaces/openshift-lws-operator/operators.coreos.com/installplans/install-zc4vr.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147331281Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147360081Z namespaces/openshift-lws-operator/operators.coreos.com/operatorconditions/leader-worker-set.v1.0.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147416053Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147455944Z namespaces/openshift-lws-operator/operators.coreos.com/operatorgroups/leader-worker-set.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147501615Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147543796Z namespaces/openshift-lws-operator/operators.coreos.com/subscriptions/leader-worker-set.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147603607Z namespaces/openshift-lws-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147611097Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147645688Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/lws-controller-manager-55f5fcc78b-5zh2r.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14772617Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14773258Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147741061Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.147780352Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148057528Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14812251Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-5zh2r/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148161791Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148203422Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/lws-controller-manager-55f5fcc78b-k2lxv.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148283094Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148292234Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/manager/manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148295654Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/manager/manager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148330505Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/manager/manager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148433937Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/manager/manager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148503209Z namespaces/openshift-lws-operator/pods/lws-controller-manager-55f5fcc78b-k2lxv/manager/manager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14854337Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148583171Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator-fd8ccff4c-ktmkv.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148661463Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148667553Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator/openshift-lws-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148672943Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator/openshift-lws-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148716114Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator/openshift-lws-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148882328Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator/openshift-lws-operator/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.14894843Z namespaces/openshift-lws-operator/pods/openshift-lws-operator-fd8ccff4c-ktmkv/openshift-lws-operator/openshift-lws-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.148984771Z namespaces/openshift-lws-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149014562Z namespaces/openshift-lws-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149155745Z namespaces/openshift-lws-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149194326Z namespaces/openshift-lws-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149236107Z namespaces/openshift-machine-api/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149242327Z namespaces/openshift-machine-api/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149245807Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149289568Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149435992Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149608236Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149747059Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.149958694Z namespaces/openshift-machine-api/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150169819Z namespaces/openshift-machine-config-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150179Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1501826Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150214941Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-controller.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150318493Z namespaces/openshift-machine-config-operator/monitoring.coreos.com/prometheusrules/machine-config-daemon.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150391155Z namespaces/openshift-machine-config-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150399215Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150436936Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.15058548Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150756914Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.150906297Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151132593Z namespaces/openshift-machine-config-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151322967Z namespaces/openshift-marketplace/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151329498Z namespaces/openshift-marketplace/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151332908Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151370509Z namespaces/openshift-marketplace/monitoring.coreos.com/prometheusrules/marketplace-alert-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.15143768Z namespaces/openshift-marketplace/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.15144411Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151475651Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/certified-operators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151570073Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/community-operators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151646265Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-marketplace.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151731817Z namespaces/openshift-marketplace/operators.coreos.com/catalogsources/redhat-operators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151786099Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.15183761Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.151988864Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152198459Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152336722Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152531697Z namespaces/openshift-marketplace/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152735272Z namespaces/openshift-monitoring/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152784923Z namespaces/openshift-monitoring/openshift-monitoring.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152850515Z namespaces/openshift-monitoring/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152923436Z namespaces/openshift-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.152987178Z namespaces/openshift-monitoring/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.153046489Z namespaces/openshift-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.153194483Z namespaces/openshift-monitoring/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.153621733Z namespaces/openshift-monitoring/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.153997452Z namespaces/openshift-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154244329Z namespaces/openshift-monitoring/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154288099Z namespaces/openshift-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154336071Z namespaces/openshift-monitoring/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154375902Z namespaces/openshift-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154452974Z namespaces/openshift-monitoring/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154518575Z namespaces/openshift-monitoring/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154562646Z namespaces/openshift-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154633998Z namespaces/openshift-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.154682679Z namespaces/openshift-monitoring/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.15471957Z namespaces/openshift-monitoring/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.157836925Z namespaces/openshift-monitoring/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.157980769Z namespaces/openshift-monitoring/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.159073055Z namespaces/openshift-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.159149987Z namespaces/openshift-monitoring/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.160047309Z namespaces/openshift-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.160182393Z namespaces/openshift-monitoring/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.160648234Z namespaces/openshift-monitoring/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.160805458Z namespaces/openshift-monitoring/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.160837429Z namespaces/openshift-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.160989252Z namespaces/openshift-monitoring/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161042454Z namespaces/openshift-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161093665Z namespaces/openshift-monitoring/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161123685Z namespaces/openshift-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161213218Z namespaces/openshift-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161287259Z namespaces/openshift-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1613289Z namespaces/openshift-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161365811Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161492524Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161524585Z namespaces/openshift-monitoring/monitoring.coreos.com/alertmanagers/main.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161604247Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161640448Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheuses/k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161762871Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161802002Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/alertmanager-main-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.161904464Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/cluster-monitoring-operator-prometheus-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.162089129Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kube-state-metrics-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.162172251Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/kubernetes-monitoring-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.162417907Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/node-exporter-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16255402Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-prometheus-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.162676333Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-k8s-thanos-sidecar-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.162764945Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/prometheus-operator-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.162880228Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/telemetry.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16296445Z namespaces/openshift-monitoring/monitoring.coreos.com/prometheusrules/thanos-querier.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163068903Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163107593Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/alertmanager-main.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163192636Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/cluster-monitoring-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163272928Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kube-state-metrics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16335981Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/kubelet.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163458102Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/metrics-server.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163537784Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/node-exporter.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163685767Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/openshift-state-metrics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16377647Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163861302Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/prometheus-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.163945744Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/telemeter-client.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164040506Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-querier.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164120908Z namespaces/openshift-monitoring/monitoring.coreos.com/servicemonitors/thanos-sidecar.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16419651Z namespaces/openshift-monitoring/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164235001Z namespaces/openshift-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164285312Z namespaces/openshift-monitoring/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164295072Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164329413Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164531378Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164742083Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.164884427Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165155803Z namespaces/openshift-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165382529Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16542491Z namespaces/openshift-monitoring/operators.coreos.com/operatorgroups/openshift-cluster-monitoring.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165504732Z namespaces/openshift-monitoring/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165512752Z namespaces/openshift-monitoring/pods/alertmanager-main-0/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165587714Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager-main-0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165747937Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165758968Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165765108Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165790289Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165883681Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165953982Z namespaces/openshift-monitoring/pods/alertmanager-main-0/alertmanager/alertmanager/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165970713Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165982073Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.165986303Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166045265Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166148027Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166208269Z namespaces/openshift-monitoring/pods/alertmanager-main-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16625041Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16625851Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16626283Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166306091Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166388343Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166451025Z namespaces/openshift-monitoring/pods/alertmanager-main-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166488006Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166494566Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166500056Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166532887Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166623769Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16668736Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-metric/kube-rbac-proxy-metric/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166727731Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166733621Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166737021Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166776533Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166899436Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166962127Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.166992858Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167001018Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167004648Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16707023Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167149972Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167216993Z namespaces/openshift-monitoring/pods/alertmanager-main-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167253674Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167259834Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167265064Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167305075Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167391817Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167449759Z namespaces/openshift-monitoring/pods/alertmanager-main-0/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1674962Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167530701Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator-54cb84584-6xhzb.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167614183Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167622173Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator/cluster-monitoring-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167627893Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator/cluster-monitoring-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.167673834Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator/cluster-monitoring-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168479534Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168545345Z namespaces/openshift-monitoring/pods/cluster-monitoring-operator-54cb84584-6xhzb/cluster-monitoring-operator/cluster-monitoring-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168575946Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168621577Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics-845b954455-wglvf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16872123Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-main/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16872746Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16874072Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168777441Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168863203Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168928565Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168964906Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-self/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168970866Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.168974106Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169008147Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169114149Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169179501Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169210672Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169216712Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics/kube-state-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169223782Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics/kube-state-metrics/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169263453Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics/kube-state-metrics/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169445877Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics/kube-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169511539Z namespaces/openshift-monitoring/pods/kube-state-metrics-845b954455-wglvf/kube-state-metrics/kube-state-metrics/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16954756Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169588091Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server-8c7b78f5c-mh7jf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169685493Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169691393Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server/metrics-server/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169695003Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server/metrics-server/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169734784Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server/metrics-server/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169822406Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server/metrics-server/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169887978Z namespaces/openshift-monitoring/pods/metrics-server-8c7b78f5c-mh7jf/metrics-server/metrics-server/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.169923129Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.16996277Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin-cd8656b8-mnnqk.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170055102Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170069702Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin/monitoring-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170073142Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin/monitoring-plugin/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170119423Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin/monitoring-plugin/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170208706Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin/monitoring-plugin/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170276247Z namespaces/openshift-monitoring/pods/monitoring-plugin-cd8656b8-mnnqk/monitoring-plugin/monitoring-plugin/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170306428Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170358689Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter-22rcx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170448371Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/init-textfile/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170461532Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/init-textfile/init-textfile/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170470612Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/init-textfile/init-textfile/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170507993Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170586875Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170646596Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170694928Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170703678Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170712158Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170745019Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170831491Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170913703Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170958234Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170965044Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter/node-exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.170970604Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter/node-exporter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171010135Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171156849Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17122633Z namespaces/openshift-monitoring/pods/node-exporter-22rcx/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171284052Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171326073Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter-f7ddq.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171413835Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/init-textfile/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171425415Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/init-textfile/init-textfile/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171431806Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/init-textfile/init-textfile/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171467276Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171537048Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171600679Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17162854Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17163634Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171651611Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171680632Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171764594Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171832585Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171873046Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171879746Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter/node-exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171882996Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter/node-exporter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.171912997Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17201649Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172104382Z namespaces/openshift-monitoring/pods/node-exporter-f7ddq/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172145163Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172180433Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter-fzt68.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172275356Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/init-textfile/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172284186Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/init-textfile/init-textfile/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172289586Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/init-textfile/init-textfile/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172320877Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/init-textfile/init-textfile/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172389709Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/init-textfile/init-textfile/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17245642Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/init-textfile/init-textfile/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172498011Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172505361Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172510482Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172540142Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172628015Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172690036Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172732877Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172738937Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter/node-exporter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172742327Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter/node-exporter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172782108Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter/node-exporter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172883641Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter/node-exporter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172949492Z namespaces/openshift-monitoring/pods/node-exporter-fzt68/node-exporter/node-exporter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.172987543Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173046975Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics-57f99c8b49-n6jk2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173216579Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-main/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173224639Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-main/kube-rbac-proxy-main/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173228439Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17326639Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173354342Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173420304Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-main/kube-rbac-proxy-main/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173459705Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-self/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173465765Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-self/kube-rbac-proxy-self/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173469495Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173507496Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173590988Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17365666Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/kube-rbac-proxy-self/kube-rbac-proxy-self/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173705231Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173714411Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics/openshift-state-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173724921Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics/openshift-state-metrics/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173761282Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics/openshift-state-metrics/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173861534Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics/openshift-state-metrics/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173925576Z namespaces/openshift-monitoring/pods/openshift-state-metrics-57f99c8b49-n6jk2/openshift-state-metrics/openshift-state-metrics/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.173999018Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17408874Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus-k8s-0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174269464Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174277044Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174282765Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174323185Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174420128Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174483389Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/config-reloader/config-reloader/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17452865Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174535271Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174545541Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174581472Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174664094Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174736156Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/init-config-reloader/init-config-reloader/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174766526Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174772616Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174776077Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174811848Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17491702Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.174981962Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-thanos/kube-rbac-proxy-thanos/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175016282Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175022193Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175040823Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175084004Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175165506Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175234288Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175273099Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175285309Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175289729Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1753199Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175398662Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175465683Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175502874Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175510054Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175515634Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.175554035Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176231272Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176300084Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/prometheus/prometheus/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176324904Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176331954Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176341494Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176383726Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176476328Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176536249Z namespaces/openshift-monitoring/pods/prometheus-k8s-0/thanos-sidecar/thanos-sidecar/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17658064Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176617121Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator-7b7d896497-5rmm2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176705303Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176713774Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176718034Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176751835Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176831146Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176898268Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176936139Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.176943269Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator/prometheus-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17695546Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator/prometheus-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17698551Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator/prometheus-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177266677Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator/prometheus-operator/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177329738Z namespaces/openshift-monitoring/pods/prometheus-operator-7b7d896497-5rmm2/prometheus-operator/prometheus-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17736803Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177406571Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177512793Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177520023Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177533243Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177573724Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177660717Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177721838Z namespaces/openshift-monitoring/pods/prometheus-operator-admission-webhook-7ddc8784d4-gh4cp/prometheus-operator-admission-webhook/prometheus-operator-admission-webhook/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177759899Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17779123Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client-789f894974-2v8t6.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177941703Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177947653Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177952904Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.177990794Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178094737Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178164029Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17820568Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/reload/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17821456Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/reload/reload/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1782352Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/reload/reload/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178263011Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/reload/reload/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178350783Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/reload/reload/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178415135Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/reload/reload/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178448326Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178454516Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client/telemeter-client/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178457946Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client/telemeter-client/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178499547Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client/telemeter-client/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17863232Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client/telemeter-client/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178690771Z namespaces/openshift-monitoring/pods/telemeter-client-789f894974-2v8t6/telemeter-client/telemeter-client/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178719642Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178791314Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-querier-5d97ff7b86-c82n6.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178931488Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178938117Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178941658Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.178977458Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179088191Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179154803Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-metrics/kube-rbac-proxy-metrics/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179198974Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-rules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179205004Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-rules/kube-rbac-proxy-rules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179208384Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179249945Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179334407Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179400039Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-rules/kube-rbac-proxy-rules/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17943516Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-web/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17944868Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-web/kube-rbac-proxy-web/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.17945391Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179479731Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179579473Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179646135Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy-web/kube-rbac-proxy-web/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179680576Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179707216Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179713096Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179754648Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179839039Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179903851Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179939652Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/prom-label-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179945462Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/prom-label-proxy/prom-label-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179948932Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/prom-label-proxy/prom-label-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.179985763Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/prom-label-proxy/prom-label-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180091466Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/prom-label-proxy/prom-label-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180159197Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/prom-label-proxy/prom-label-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180195198Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-query/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180201348Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-query/thanos-query/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180211419Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-query/thanos-query/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1802489Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-query/thanos-query/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180348522Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-query/thanos-query/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180412653Z namespaces/openshift-monitoring/pods/thanos-querier-5d97ff7b86-c82n6/thanos-query/thanos-query/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180445034Z namespaces/openshift-monitoring/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180484615Z namespaces/openshift-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180535986Z namespaces/openshift-monitoring/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180573227Z namespaces/openshift-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18066649Z namespaces/openshift-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180705971Z namespaces/openshift-multus/openshift-multus.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180779232Z namespaces/openshift-multus/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180816873Z namespaces/openshift-multus/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180864684Z namespaces/openshift-multus/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.180904245Z namespaces/openshift-multus/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181168692Z namespaces/openshift-multus/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181242273Z namespaces/openshift-multus/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181316495Z namespaces/openshift-multus/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181360826Z namespaces/openshift-multus/apps/daemonsets/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181401227Z namespaces/openshift-multus/apps/daemonsets/multus-additional-cni-plugins.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181544101Z namespaces/openshift-multus/apps/daemonsets/multus.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181657534Z namespaces/openshift-multus/apps/daemonsets/network-metrics-daemon.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181717315Z namespaces/openshift-multus/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181757946Z namespaces/openshift-multus/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181809027Z namespaces/openshift-multus/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181855808Z namespaces/openshift-multus/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18192532Z namespaces/openshift-multus/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.181974131Z namespaces/openshift-multus/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182016202Z namespaces/openshift-multus/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182112305Z namespaces/openshift-multus/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182175696Z namespaces/openshift-multus/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182219737Z namespaces/openshift-multus/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182350691Z namespaces/openshift-multus/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182431432Z namespaces/openshift-multus/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.182900004Z namespaces/openshift-multus/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.183017496Z namespaces/openshift-multus/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.183661202Z namespaces/openshift-multus/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.183749614Z namespaces/openshift-multus/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.183876528Z namespaces/openshift-multus/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.183924429Z namespaces/openshift-multus/core/configmaps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18395663Z namespaces/openshift-multus/core/configmaps/cni-copy-resources.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184078842Z namespaces/openshift-multus/core/configmaps/default-cni-sysctl-allowlist.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184164304Z namespaces/openshift-multus/core/configmaps/multus-daemon-config.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184253377Z namespaces/openshift-multus/core/configmaps/whereabouts-flatfile-config.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184341739Z namespaces/openshift-multus/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18438715Z namespaces/openshift-multus/core/serviceaccounts/metrics-daemon-sa.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184476652Z namespaces/openshift-multus/core/serviceaccounts/multus-ac.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184557844Z namespaces/openshift-multus/core/serviceaccounts/multus-ancillary-tools.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184633756Z namespaces/openshift-multus/core/serviceaccounts/multus.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184684727Z namespaces/openshift-multus/core/services/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184727758Z namespaces/openshift-multus/core/services/network-metrics-service.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18478693Z namespaces/openshift-multus/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18482984Z namespaces/openshift-multus/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184881152Z namespaces/openshift-multus/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184922593Z namespaces/openshift-multus/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.184974764Z namespaces/openshift-multus/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185016715Z namespaces/openshift-multus/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185121278Z namespaces/openshift-multus/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18521579Z namespaces/openshift-multus/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185269101Z namespaces/openshift-multus/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185314302Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185359003Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185406874Z namespaces/openshift-multus/monitoring.coreos.com/servicemonitors/monitor-network.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185452916Z namespaces/openshift-multus/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185496677Z namespaces/openshift-multus/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185536938Z namespaces/openshift-multus/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185551048Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185602499Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185754083Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.185933787Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186093711Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186296496Z namespaces/openshift-multus/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186481531Z namespaces/openshift-multus/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186488251Z namespaces/openshift-multus/pods/multus-85vx9/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186527552Z namespaces/openshift-multus/pods/multus-85vx9/multus-85vx9.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186623644Z namespaces/openshift-multus/pods/multus-85vx9/kube-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186630094Z namespaces/openshift-multus/pods/multus-85vx9/kube-multus/kube-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186633544Z namespaces/openshift-multus/pods/multus-85vx9/kube-multus/kube-multus/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.186677975Z namespaces/openshift-multus/pods/multus-85vx9/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187541326Z namespaces/openshift-multus/pods/multus-85vx9/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187604728Z namespaces/openshift-multus/pods/multus-85vx9/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187655459Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18769372Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/multus-additional-cni-plugins-qbnbc.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187874214Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/bond-cni-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187881755Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187887655Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.187913955Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188010068Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18811022Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188134151Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188139271Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/cni-plugins/cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188153241Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188241563Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188341686Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188408257Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188473219Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/egress-router-binary-copy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188480029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188483469Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188550391Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188649193Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188717465Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188749986Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188756416Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188763546Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188811587Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188879099Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18894746Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188980551Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/routeoverride-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188987441Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.188991332Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189049483Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189215027Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189312869Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18934563Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni-bincopy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.18935688Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189362141Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189394371Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189479133Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189550525Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189585996Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189594676Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189600026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189626757Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189716149Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189784401Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-qbnbc/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189820122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189866163Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/multus-additional-cni-plugins-vrktm.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189991026Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/bond-cni-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.189998056Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190002216Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190056438Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19015495Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190221891Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190260842Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190268292Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/cni-plugins/cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190271663Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190305864Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190393166Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190464047Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190499958Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/egress-router-binary-copy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190509148Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190514189Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190551159Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190632661Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190700833Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190735584Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190744354Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190749334Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190780545Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190857907Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190932849Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.190969309Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/routeoverride-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19097637Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19097967Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191012701Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191117193Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191182915Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191221566Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni-bincopy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191230286Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191240046Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191270777Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191356499Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191425731Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191461952Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191468122Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191476402Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191511583Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191595075Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191664066Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-vrktm/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191728888Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191776909Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/multus-additional-cni-plugins-xb4ww.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191908132Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/bond-cni-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191915392Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/bond-cni-plugin/bond-cni-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191919443Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/bond-cni-plugin/bond-cni-plugin/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.191954264Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/bond-cni-plugin/bond-cni-plugin/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192059636Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/bond-cni-plugin/bond-cni-plugin/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192131138Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/bond-cni-plugin/bond-cni-plugin/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192167759Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192176829Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/cni-plugins/cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192182029Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/cni-plugins/cni-plugins/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19220837Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/cni-plugins/cni-plugins/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192293102Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/cni-plugins/cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192362313Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/cni-plugins/cni-plugins/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192394924Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/egress-router-binary-copy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192403884Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/egress-router-binary-copy/egress-router-binary-copy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192408524Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/egress-router-binary-copy/egress-router-binary-copy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192437295Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/egress-router-binary-copy/egress-router-binary-copy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192519937Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/egress-router-binary-copy/egress-router-binary-copy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192587749Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/egress-router-binary-copy/egress-router-binary-copy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.1926229Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19262904Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19263479Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192666651Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192735312Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192801604Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/kube-multus-additional-cni-plugins/kube-multus-additional-cni-plugins/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192838565Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/routeoverride-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192845585Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/routeoverride-cni/routeoverride-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192848925Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/routeoverride-cni/routeoverride-cni/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192878486Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/routeoverride-cni/routeoverride-cni/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.192965778Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/routeoverride-cni/routeoverride-cni/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19305201Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/routeoverride-cni/routeoverride-cni/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193157042Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni-bincopy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193163683Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni-bincopy/whereabouts-cni-bincopy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193171813Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193195754Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193284296Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193350547Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni-bincopy/whereabouts-cni-bincopy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193378158Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193385818Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni/whereabouts-cni/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193391308Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni/whereabouts-cni/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193420609Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni/whereabouts-cni/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193502111Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni/whereabouts-cni/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193571593Z namespaces/openshift-multus/pods/multus-additional-cni-plugins-xb4ww/whereabouts-cni/whereabouts-cni/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193596663Z namespaces/openshift-multus/pods/multus-ksggx/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193639644Z namespaces/openshift-multus/pods/multus-ksggx/multus-ksggx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193729807Z namespaces/openshift-multus/pods/multus-ksggx/kube-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193742277Z namespaces/openshift-multus/pods/multus-ksggx/kube-multus/kube-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193746697Z namespaces/openshift-multus/pods/multus-ksggx/kube-multus/kube-multus/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.193766537Z namespaces/openshift-multus/pods/multus-ksggx/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194541696Z namespaces/openshift-multus/pods/multus-ksggx/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194609708Z namespaces/openshift-multus/pods/multus-ksggx/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194646679Z namespaces/openshift-multus/pods/multus-m8mqn/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194682099Z namespaces/openshift-multus/pods/multus-m8mqn/multus-m8mqn.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194766622Z namespaces/openshift-multus/pods/multus-m8mqn/kube-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194777602Z namespaces/openshift-multus/pods/multus-m8mqn/kube-multus/kube-multus/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194781382Z namespaces/openshift-multus/pods/multus-m8mqn/kube-multus/kube-multus/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.194821803Z namespaces/openshift-multus/pods/multus-m8mqn/kube-multus/kube-multus/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195323035Z namespaces/openshift-multus/pods/multus-m8mqn/kube-multus/kube-multus/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195382227Z namespaces/openshift-multus/pods/multus-m8mqn/kube-multus/kube-multus/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195405297Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195451528Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon-cthwx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19553396Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19554012Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195552291Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195582752Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195686284Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195752816Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195791217Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195799517Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195803267Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195844718Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.195979811Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196069783Z namespaces/openshift-multus/pods/network-metrics-daemon-cthwx/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196090394Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196141735Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon-sqdk2.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196214857Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196221037Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196224467Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196263658Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19636188Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196428582Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196451213Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196463673Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196474243Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196519824Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196699419Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19676529Z namespaces/openshift-multus/pods/network-metrics-daemon-sqdk2/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196792901Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196837152Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon-xw4rh.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196910854Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196919554Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/kube-rbac-proxy/kube-rbac-proxy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196925194Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/kube-rbac-proxy/kube-rbac-proxy/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.196952845Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/kube-rbac-proxy/kube-rbac-proxy/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197057857Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197129589Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/kube-rbac-proxy/kube-rbac-proxy/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19716972Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19718405Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon/network-metrics-daemon/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19718953Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon/network-metrics-daemon/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197232141Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon/network-metrics-daemon/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197399416Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon/network-metrics-daemon/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197466147Z namespaces/openshift-multus/pods/network-metrics-daemon-xw4rh/network-metrics-daemon/network-metrics-daemon/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197502448Z namespaces/openshift-multus/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197539249Z namespaces/openshift-multus/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19758353Z namespaces/openshift-multus/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19758969Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197624621Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/multus-whereabouts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197710613Z namespaces/openshift-multus/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197760234Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197794635Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197873107Z namespaces/openshift-multus/rbac.authorization.k8s.io/roles/whereabouts-cni.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197920608Z namespaces/openshift-multus/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.197956439Z namespaces/openshift-multus/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19800386Z namespaces/openshift-must-gather-z7v9q/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19800977Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19801496Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.198077782Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.198235266Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19840942Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.198553624Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.198760259Z namespaces/openshift-must-gather-z7v9q/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.198957594Z namespaces/openshift-network-console/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.198998055Z namespaces/openshift-network-console/openshift-network-console.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199128727Z namespaces/openshift-network-console/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199168888Z namespaces/openshift-network-console/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199204049Z namespaces/openshift-network-console/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19925347Z namespaces/openshift-network-console/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199335152Z namespaces/openshift-network-console/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199459415Z namespaces/openshift-network-console/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199558438Z namespaces/openshift-network-console/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199591669Z namespaces/openshift-network-console/apps/deployments/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.19963986Z namespaces/openshift-network-console/apps/deployments/networking-console-plugin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199713992Z namespaces/openshift-network-console/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199753573Z namespaces/openshift-network-console/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199813754Z namespaces/openshift-network-console/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199827335Z namespaces/openshift-network-console/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199911877Z namespaces/openshift-network-console/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199956118Z namespaces/openshift-network-console/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.199991739Z namespaces/openshift-network-console/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200099291Z namespaces/openshift-network-console/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200136012Z namespaces/openshift-network-console/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200177663Z namespaces/openshift-network-console/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200289606Z namespaces/openshift-network-console/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200367557Z namespaces/openshift-network-console/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200486271Z namespaces/openshift-network-console/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200561602Z namespaces/openshift-network-console/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200703426Z namespaces/openshift-network-console/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200827889Z namespaces/openshift-network-console/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.200942071Z namespaces/openshift-network-console/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201000823Z namespaces/openshift-network-console/core/configmaps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201066254Z namespaces/openshift-network-console/core/configmaps/networking-console-plugin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201116266Z namespaces/openshift-network-console/core/services/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201161977Z namespaces/openshift-network-console/core/services/networking-console-plugin.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201205888Z namespaces/openshift-network-console/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201245239Z namespaces/openshift-network-console/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20130152Z namespaces/openshift-network-console/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201349041Z namespaces/openshift-network-console/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201383452Z namespaces/openshift-network-console/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201422103Z namespaces/openshift-network-console/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201500475Z namespaces/openshift-network-console/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201575937Z namespaces/openshift-network-console/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201638299Z namespaces/openshift-network-console/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20168196Z namespaces/openshift-network-console/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201721621Z namespaces/openshift-network-console/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201760991Z namespaces/openshift-network-console/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201806443Z namespaces/openshift-network-console/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201814303Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201844784Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.201990267Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202190372Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202326935Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20252516Z namespaces/openshift-network-console/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202710934Z namespaces/openshift-network-console/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202717435Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202747505Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin-6cdb5968b9-fhkqt.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202825577Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202831988Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin/networking-console-plugin/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202835437Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin/networking-console-plugin/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202885349Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin/networking-console-plugin/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.202974591Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin/networking-console-plugin/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203039102Z namespaces/openshift-network-console/pods/networking-console-plugin-6cdb5968b9-fhkqt/networking-console-plugin/networking-console-plugin/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203134575Z namespaces/openshift-network-console/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203165125Z namespaces/openshift-network-console/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203216407Z namespaces/openshift-network-console/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203257308Z namespaces/openshift-network-console/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203303569Z namespaces/openshift-network-diagnostics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20334874Z namespaces/openshift-network-diagnostics/openshift-network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203403231Z namespaces/openshift-network-diagnostics/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203442112Z namespaces/openshift-network-diagnostics/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203486223Z namespaces/openshift-network-diagnostics/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203526124Z namespaces/openshift-network-diagnostics/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203628237Z namespaces/openshift-network-diagnostics/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203723869Z namespaces/openshift-network-diagnostics/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203817511Z namespaces/openshift-network-diagnostics/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203858362Z namespaces/openshift-network-diagnostics/apps/daemonsets/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203899643Z namespaces/openshift-network-diagnostics/apps/daemonsets/network-check-target.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.203959055Z namespaces/openshift-network-diagnostics/apps/deployments/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204006166Z namespaces/openshift-network-diagnostics/apps/deployments/network-check-source.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204094158Z namespaces/openshift-network-diagnostics/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204121909Z namespaces/openshift-network-diagnostics/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20417261Z namespaces/openshift-network-diagnostics/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204214371Z namespaces/openshift-network-diagnostics/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204293563Z namespaces/openshift-network-diagnostics/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204334224Z namespaces/openshift-network-diagnostics/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204373125Z namespaces/openshift-network-diagnostics/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204449627Z namespaces/openshift-network-diagnostics/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204492078Z namespaces/openshift-network-diagnostics/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204535249Z namespaces/openshift-network-diagnostics/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204625951Z namespaces/openshift-network-diagnostics/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204712193Z namespaces/openshift-network-diagnostics/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204893487Z namespaces/openshift-network-diagnostics/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.204973419Z namespaces/openshift-network-diagnostics/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205190095Z namespaces/openshift-network-diagnostics/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205272567Z namespaces/openshift-network-diagnostics/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205367309Z namespaces/openshift-network-diagnostics/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20542303Z namespaces/openshift-network-diagnostics/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205467981Z namespaces/openshift-network-diagnostics/core/serviceaccounts/network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205515702Z namespaces/openshift-network-diagnostics/core/services/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205556854Z namespaces/openshift-network-diagnostics/core/services/network-check-source.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205636075Z namespaces/openshift-network-diagnostics/core/services/network-check-target.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205684006Z namespaces/openshift-network-diagnostics/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205726667Z namespaces/openshift-network-diagnostics/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205785629Z namespaces/openshift-network-diagnostics/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20582783Z namespaces/openshift-network-diagnostics/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205870251Z namespaces/openshift-network-diagnostics/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205910052Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.205986684Z namespaces/openshift-network-diagnostics/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206095327Z namespaces/openshift-network-diagnostics/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206139228Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206181319Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20623Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206274451Z namespaces/openshift-network-diagnostics/monitoring.coreos.com/servicemonitors/network-check-source.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206321072Z namespaces/openshift-network-diagnostics/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206358823Z namespaces/openshift-network-diagnostics/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206405544Z namespaces/openshift-network-diagnostics/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206412444Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206451245Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206590579Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206767663Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.206912436Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207166242Z namespaces/openshift-network-diagnostics/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207346847Z namespaces/openshift-network-diagnostics/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207354677Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207386448Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/network-check-source-56499946bc-86srq.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20746233Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/check-endpoints/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20746988Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/check-endpoints/check-endpoints/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20747413Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/check-endpoints/check-endpoints/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207513141Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/check-endpoints/check-endpoints/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207623144Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/check-endpoints/check-endpoints/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207687775Z namespaces/openshift-network-diagnostics/pods/network-check-source-56499946bc-86srq/check-endpoints/check-endpoints/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207726456Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207755957Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-6gj4t.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207833939Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-container/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207841379Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207845649Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20787171Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.207965752Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208045364Z namespaces/openshift-network-diagnostics/pods/network-check-target-6gj4t/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208089195Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208128006Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-fz5ld.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208197118Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-container/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208207788Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208211778Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208247179Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208328231Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208393762Z namespaces/openshift-network-diagnostics/pods/network-check-target-fz5ld/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208425823Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208464824Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-wzfcf.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208535396Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-container/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208541296Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-container/network-check-target-container/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208544796Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-container/network-check-target-container/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208581577Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-container/network-check-target-container/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208664859Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-container/network-check-target-container/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208734171Z namespaces/openshift-network-diagnostics/pods/network-check-target-wzfcf/network-check-target-container/network-check-target-container/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208760551Z namespaces/openshift-network-diagnostics/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208805712Z namespaces/openshift-network-diagnostics/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208841803Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208848733Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208885924Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.208969876Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209016718Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209080189Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/network-diagnostics.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209164541Z namespaces/openshift-network-diagnostics/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209206872Z namespaces/openshift-network-diagnostics/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209247713Z namespaces/openshift-network-diagnostics/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209289154Z namespaces/openshift-network-node-identity/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209343925Z namespaces/openshift-network-node-identity/openshift-network-node-identity.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209393776Z namespaces/openshift-network-node-identity/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209437368Z namespaces/openshift-network-node-identity/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209479499Z namespaces/openshift-network-node-identity/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.20952041Z namespaces/openshift-network-node-identity/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209599901Z namespaces/openshift-network-node-identity/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209673213Z namespaces/openshift-network-node-identity/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209749435Z namespaces/openshift-network-node-identity/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209798856Z namespaces/openshift-network-node-identity/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209833957Z namespaces/openshift-network-node-identity/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209877878Z namespaces/openshift-network-node-identity/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209921719Z namespaces/openshift-network-node-identity/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.209996181Z namespaces/openshift-network-node-identity/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210057213Z namespaces/openshift-network-node-identity/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210101804Z namespaces/openshift-network-node-identity/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210182896Z namespaces/openshift-network-node-identity/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210224177Z namespaces/openshift-network-node-identity/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210231757Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210265678Z namespaces/openshift-network-node-identity/coordination.k8s.io/leases/ovnkube-identity.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210315039Z namespaces/openshift-network-node-identity/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2103531Z namespaces/openshift-network-node-identity/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210459882Z namespaces/openshift-network-node-identity/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210531464Z namespaces/openshift-network-node-identity/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210619796Z namespaces/openshift-network-node-identity/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210694108Z namespaces/openshift-network-node-identity/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21077381Z namespaces/openshift-network-node-identity/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.210922724Z namespaces/openshift-network-node-identity/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211019126Z namespaces/openshift-network-node-identity/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21119402Z namespaces/openshift-network-node-identity/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211248401Z namespaces/openshift-network-node-identity/core/serviceaccounts/network-node-identity.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211315363Z namespaces/openshift-network-node-identity/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211360214Z namespaces/openshift-network-node-identity/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211410025Z namespaces/openshift-network-node-identity/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211449077Z namespaces/openshift-network-node-identity/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211489847Z namespaces/openshift-network-node-identity/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211531368Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21160745Z namespaces/openshift-network-node-identity/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211681612Z namespaces/openshift-network-node-identity/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211724543Z namespaces/openshift-network-node-identity/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211763214Z namespaces/openshift-network-node-identity/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211805465Z namespaces/openshift-network-node-identity/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211845246Z namespaces/openshift-network-node-identity/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211888077Z namespaces/openshift-network-node-identity/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211894077Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.211929328Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212111942Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212284927Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21241562Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212613945Z namespaces/openshift-network-node-identity/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212793399Z namespaces/openshift-network-node-identity/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21282445Z namespaces/openshift-network-node-identity/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212875491Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212881831Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212917902Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/rolebindings/network-node-identity-leases.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.212965923Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213004364Z namespaces/openshift-network-node-identity/rbac.authorization.k8s.io/roles/network-node-identity-leases.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213084446Z namespaces/openshift-network-node-identity/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213115067Z namespaces/openshift-network-node-identity/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213166758Z namespaces/openshift-network-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213201959Z namespaces/openshift-network-operator/openshift-network-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213263961Z namespaces/openshift-network-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213301781Z namespaces/openshift-network-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213350932Z namespaces/openshift-network-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213391664Z namespaces/openshift-network-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213483516Z namespaces/openshift-network-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213557077Z namespaces/openshift-network-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21363481Z namespaces/openshift-network-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2136795Z namespaces/openshift-network-operator/apps/daemonsets/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213720172Z namespaces/openshift-network-operator/apps/daemonsets/iptables-alerter.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213779203Z namespaces/openshift-network-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213824494Z namespaces/openshift-network-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213864725Z namespaces/openshift-network-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213899276Z namespaces/openshift-network-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.213975788Z namespaces/openshift-network-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214044119Z namespaces/openshift-network-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214087661Z namespaces/openshift-network-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214171472Z namespaces/openshift-network-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214212093Z namespaces/openshift-network-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214219364Z namespaces/openshift-network-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214254165Z namespaces/openshift-network-operator/coordination.k8s.io/leases/network-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214302326Z namespaces/openshift-network-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214344967Z namespaces/openshift-network-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21449112Z namespaces/openshift-network-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214573622Z namespaces/openshift-network-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214730546Z namespaces/openshift-network-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214803458Z namespaces/openshift-network-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.214994543Z namespaces/openshift-network-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215147826Z namespaces/openshift-network-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215276369Z namespaces/openshift-network-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215324Z namespaces/openshift-network-operator/core/configmaps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215375592Z namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215452043Z namespaces/openshift-network-operator/core/configmaps/iptables-alerter-script.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215508715Z namespaces/openshift-network-operator/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215551366Z namespaces/openshift-network-operator/core/serviceaccounts/iptables-alerter.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215597347Z namespaces/openshift-network-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215643018Z namespaces/openshift-network-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215691439Z namespaces/openshift-network-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21573208Z namespaces/openshift-network-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215771201Z namespaces/openshift-network-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215812002Z namespaces/openshift-network-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215889384Z namespaces/openshift-network-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.215961746Z namespaces/openshift-network-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216007457Z namespaces/openshift-network-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216086029Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21614462Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216192041Z namespaces/openshift-network-operator/monitoring.coreos.com/prometheusrules/openshift-network-operator-ipsec-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216245063Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216283824Z namespaces/openshift-network-operator/monitoring.coreos.com/servicemonitors/network-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216333475Z namespaces/openshift-network-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216370686Z namespaces/openshift-network-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216413737Z namespaces/openshift-network-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216420657Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216457168Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216602702Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216774996Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.216908899Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217149925Z namespaces/openshift-network-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217336409Z namespaces/openshift-network-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21734448Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21736345Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter-h52h5.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217446352Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217456252Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217462432Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217483673Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217570125Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217630776Z namespaces/openshift-network-operator/pods/iptables-alerter-h52h5/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217664837Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217708258Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter-lhmmx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21777027Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21777826Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21778255Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217817081Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217897233Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217964715Z namespaces/openshift-network-operator/pods/iptables-alerter-lhmmx/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.217997405Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218063187Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter-tfqlz.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218142369Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218158799Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter/iptables-alerter/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21816544Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter/iptables-alerter/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21819747Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter/iptables-alerter/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218283702Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter/iptables-alerter/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218350794Z namespaces/openshift-network-operator/pods/iptables-alerter-tfqlz/iptables-alerter/iptables-alerter/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218383215Z namespaces/openshift-network-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218424486Z namespaces/openshift-network-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218467007Z namespaces/openshift-network-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218505758Z namespaces/openshift-network-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218551019Z namespaces/openshift-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218557639Z namespaces/openshift-node/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218561009Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.21859268Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218746954Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.218937608Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219106892Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219307977Z namespaces/openshift-node/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219488992Z namespaces/openshift-operator-lifecycle-manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219533083Z namespaces/openshift-operator-lifecycle-manager/openshift-operator-lifecycle-manager.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219593954Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219618185Z namespaces/openshift-operator-lifecycle-manager/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219670106Z namespaces/openshift-operator-lifecycle-manager/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219701637Z namespaces/openshift-operator-lifecycle-manager/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219783589Z namespaces/openshift-operator-lifecycle-manager/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219856641Z namespaces/openshift-operator-lifecycle-manager/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219933212Z namespaces/openshift-operator-lifecycle-manager/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.219979213Z namespaces/openshift-operator-lifecycle-manager/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220043265Z namespaces/openshift-operator-lifecycle-manager/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220100767Z namespaces/openshift-operator-lifecycle-manager/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220146228Z namespaces/openshift-operator-lifecycle-manager/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220222119Z namespaces/openshift-operator-lifecycle-manager/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220273481Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220307161Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220381413Z namespaces/openshift-operator-lifecycle-manager/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220426734Z namespaces/openshift-operator-lifecycle-manager/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220469465Z namespaces/openshift-operator-lifecycle-manager/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220567088Z namespaces/openshift-operator-lifecycle-manager/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22064622Z namespaces/openshift-operator-lifecycle-manager/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220727582Z namespaces/openshift-operator-lifecycle-manager/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220805104Z namespaces/openshift-operator-lifecycle-manager/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220876445Z namespaces/openshift-operator-lifecycle-manager/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.220998108Z namespaces/openshift-operator-lifecycle-manager/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221228544Z namespaces/openshift-operator-lifecycle-manager/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221290275Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221322046Z namespaces/openshift-operator-lifecycle-manager/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221385627Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221420799Z namespaces/openshift-operator-lifecycle-manager/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221467249Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221508121Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221578502Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221656304Z namespaces/openshift-operator-lifecycle-manager/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221695025Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221736896Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221779787Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221821868Z namespaces/openshift-operator-lifecycle-manager/monitoring.coreos.com/prometheusrules/olm-alert-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22187869Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2219154Z namespaces/openshift-operator-lifecycle-manager/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221978942Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.221986242Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222043634Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222209558Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222380162Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222512545Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222703299Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222900604Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222932485Z namespaces/openshift-operator-lifecycle-manager/operators.coreos.com/operatorgroups/olm-operators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.222992826Z namespaces/openshift-operator-lifecycle-manager/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223047148Z namespaces/openshift-operator-lifecycle-manager/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223107149Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22315192Z namespaces/openshift-operator-lifecycle-manager/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223193851Z namespaces/openshift-operators/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223201242Z namespaces/openshift-operators/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223205322Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223231342Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223378436Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22355098Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223683393Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.223874978Z namespaces/openshift-operators/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224086093Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224110894Z namespaces/openshift-operators/operators.coreos.com/operatorgroups/global-operators.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224171475Z namespaces/openshift-ovn-kubernetes/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224207926Z namespaces/openshift-ovn-kubernetes/openshift-ovn-kubernetes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224278838Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224298768Z namespaces/openshift-ovn-kubernetes/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22435577Z namespaces/openshift-ovn-kubernetes/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22439476Z namespaces/openshift-ovn-kubernetes/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224562545Z namespaces/openshift-ovn-kubernetes/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224635316Z namespaces/openshift-ovn-kubernetes/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224714138Z namespaces/openshift-ovn-kubernetes/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22476074Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224801201Z namespaces/openshift-ovn-kubernetes/apps/daemonsets/ovnkube-node.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224947464Z namespaces/openshift-ovn-kubernetes/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.224987775Z namespaces/openshift-ovn-kubernetes/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225062797Z namespaces/openshift-ovn-kubernetes/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225108398Z namespaces/openshift-ovn-kubernetes/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22518837Z namespaces/openshift-ovn-kubernetes/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225230671Z namespaces/openshift-ovn-kubernetes/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225271792Z namespaces/openshift-ovn-kubernetes/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225342834Z namespaces/openshift-ovn-kubernetes/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225390305Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225396455Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225433476Z namespaces/openshift-ovn-kubernetes/coordination.k8s.io/leases/ovn-kubernetes-master.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225482557Z namespaces/openshift-ovn-kubernetes/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225518758Z namespaces/openshift-ovn-kubernetes/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225709872Z namespaces/openshift-ovn-kubernetes/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.225792405Z namespaces/openshift-ovn-kubernetes/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.226138443Z namespaces/openshift-ovn-kubernetes/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.226256806Z namespaces/openshift-ovn-kubernetes/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.226738047Z namespaces/openshift-ovn-kubernetes/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.226809259Z namespaces/openshift-ovn-kubernetes/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.226981203Z namespaces/openshift-ovn-kubernetes/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227037755Z namespaces/openshift-ovn-kubernetes/core/configmaps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227140567Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-config.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227226119Z namespaces/openshift-ovn-kubernetes/core/configmaps/ovnkube-script-lib.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227369093Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227406594Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-control-plane.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227491406Z namespaces/openshift-ovn-kubernetes/core/serviceaccounts/ovn-kubernetes-node.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227527507Z namespaces/openshift-ovn-kubernetes/core/services/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227574928Z namespaces/openshift-ovn-kubernetes/core/services/ovn-kubernetes-node.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227625449Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22765502Z namespaces/openshift-ovn-kubernetes/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227788223Z namespaces/openshift-ovn-kubernetes/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227828174Z namespaces/openshift-ovn-kubernetes/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227875215Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227885725Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227915706Z namespaces/openshift-ovn-kubernetes/k8s.cni.cncf.io/network-attachment-definitions/default.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.227969327Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228007148Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228118101Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228191533Z namespaces/openshift-ovn-kubernetes/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228240074Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228279255Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228325556Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228367907Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/prometheusrules/networking-rules.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228440269Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22847952Z namespaces/openshift-ovn-kubernetes/monitoring.coreos.com/servicemonitors/monitor-ovn-node.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228524151Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228530581Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228565822Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/ovn.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228651004Z namespaces/openshift-ovn-kubernetes/network.operator.openshift.io/operatorpkis/signer.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228694265Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228736266Z namespaces/openshift-ovn-kubernetes/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228778117Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228784387Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228822428Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.228962731Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229175836Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.22931139Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229506055Z namespaces/openshift-ovn-kubernetes/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229693229Z namespaces/openshift-ovn-kubernetes/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229700059Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229767071Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-node-lrvqz.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229935425Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229942525Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229946625Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.229976896Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230118619Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230182541Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230213702Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230218102Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230221242Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230260363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230364456Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230430927Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230463788Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/nbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230468228Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/nbdb/nbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230471228Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/nbdb/nbdb/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230504529Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/nbdb/nbdb/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230596841Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230670543Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230692553Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/northd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230696853Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/northd/northd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230701873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/northd/northd/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230737994Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/northd/northd/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230840857Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.230923969Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/northd/northd/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23095795Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-acl-logging/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23096568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23096985Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231004141Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231131854Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231181315Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231189195Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-controller/ovn-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231192665Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231231457Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231529574Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231595975Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231629766Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231636916Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231640286Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.231660607Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233483591Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233549953Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233583174Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/sbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233589334Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/sbdb/sbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233592974Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/sbdb/sbdb/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233625024Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/sbdb/sbdb/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233719387Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233785448Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-lrvqz/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233808649Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.233882681Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-node-q2zrq.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234044555Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234056445Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234060535Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234102886Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234212339Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23427338Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234314321Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234320721Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234324681Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234354512Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234460375Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234527796Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234562117Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/nbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234568517Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/nbdb/nbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234571887Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/nbdb/nbdb/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234601908Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/nbdb/nbdb/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23469453Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234760182Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234787523Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/northd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234793363Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/northd/northd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234796683Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/northd/northd/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234837504Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/northd/northd/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.234945767Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235013848Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/northd/northd/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235122151Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-acl-logging/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235133671Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235139361Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235170002Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235259194Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235308315Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235314476Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-controller/ovn-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235317916Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235356257Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235658394Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235723815Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235758396Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235763116Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235766527Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.235804047Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237526239Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237591141Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237629592Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/sbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237639122Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/sbdb/sbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237643302Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/sbdb/sbdb/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237669723Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/sbdb/sbdb/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237760565Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237825757Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-q2zrq/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237862777Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.237926049Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-node-sw9rx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238094983Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238106063Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-node/kube-rbac-proxy-node/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238110063Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238123084Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238236747Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238304288Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-node/kube-rbac-proxy-node/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238336429Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238343759Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238347069Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23838Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238501973Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238563364Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/kube-rbac-proxy-ovn-metrics/kube-rbac-proxy-ovn-metrics/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238601425Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/nbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238608905Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/nbdb/nbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238613026Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/nbdb/nbdb/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238637316Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/nbdb/nbdb/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238730779Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/nbdb/nbdb/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.23879379Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/nbdb/nbdb/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238832691Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/northd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238838801Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/northd/northd/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238842011Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/northd/northd/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238884962Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/northd/northd/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.238991925Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/northd/northd/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239089547Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/northd/northd/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239103737Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-acl-logging/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239109568Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-acl-logging/ovn-acl-logging/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239115198Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-acl-logging/ovn-acl-logging/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239158259Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-acl-logging/ovn-acl-logging/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239245931Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-acl-logging/ovn-acl-logging/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239319753Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-acl-logging/ovn-acl-logging/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239345863Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239359034Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-controller/ovn-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239363314Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-controller/ovn-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239394264Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-controller/ovn-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2396129Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-controller/ovn-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239678231Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovn-controller/ovn-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239702622Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239708932Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-controller/ovnkube-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239712642Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-controller/ovnkube-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.239750933Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-controller/ovnkube-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241180148Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-controller/ovnkube-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241247029Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/ovnkube-controller/ovnkube-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24126248Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/sbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24126873Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/sbdb/sbdb/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24127198Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/sbdb/sbdb/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241319051Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/sbdb/sbdb/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241421674Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/sbdb/sbdb/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241477205Z namespaces/openshift-ovn-kubernetes/pods/ovnkube-node-sw9rx/sbdb/sbdb/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241513446Z namespaces/openshift-ovn-kubernetes/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241544047Z namespaces/openshift-ovn-kubernetes/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241589238Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241595948Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241638239Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241730771Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/openshift-ovn-kubernetes-nodes-identity-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241847324Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/rolebindings/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241897875Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.241934236Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-control-plane-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242040029Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/openshift-ovn-kubernetes-node-limited.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242141901Z namespaces/openshift-ovn-kubernetes/rbac.authorization.k8s.io/roles/prometheus-k8s.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242185832Z namespaces/openshift-ovn-kubernetes/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242229473Z namespaces/openshift-ovn-kubernetes/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242268824Z namespaces/openshift-route-controller-manager/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242275254Z namespaces/openshift-route-controller-manager/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242278854Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242311095Z namespaces/openshift-route-controller-manager/coordination.k8s.io/leases/openshift-route-controllers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242355776Z namespaces/openshift-route-controller-manager/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242361847Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242407398Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242591972Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242796127Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.242962231Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243250708Z namespaces/openshift-route-controller-manager/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243479004Z namespaces/openshift-service-ca-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243516864Z namespaces/openshift-service-ca-operator/openshift-service-ca-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243577246Z namespaces/openshift-service-ca-operator/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243619177Z namespaces/openshift-service-ca-operator/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243666578Z namespaces/openshift-service-ca-operator/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243705509Z namespaces/openshift-service-ca-operator/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243783851Z namespaces/openshift-service-ca-operator/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243882013Z namespaces/openshift-service-ca-operator/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.243980216Z namespaces/openshift-service-ca-operator/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244015857Z namespaces/openshift-service-ca-operator/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244088358Z namespaces/openshift-service-ca-operator/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244126729Z namespaces/openshift-service-ca-operator/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24417067Z namespaces/openshift-service-ca-operator/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244254102Z namespaces/openshift-service-ca-operator/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244293753Z namespaces/openshift-service-ca-operator/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244329574Z namespaces/openshift-service-ca-operator/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244406806Z namespaces/openshift-service-ca-operator/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244447687Z namespaces/openshift-service-ca-operator/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244454427Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244485738Z namespaces/openshift-service-ca-operator/coordination.k8s.io/leases/service-ca-operator-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244535299Z namespaces/openshift-service-ca-operator/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24457846Z namespaces/openshift-service-ca-operator/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244698343Z namespaces/openshift-service-ca-operator/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.244779565Z namespaces/openshift-service-ca-operator/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24496091Z namespaces/openshift-service-ca-operator/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245053982Z namespaces/openshift-service-ca-operator/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245164895Z namespaces/openshift-service-ca-operator/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245277907Z namespaces/openshift-service-ca-operator/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245446941Z namespaces/openshift-service-ca-operator/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245499723Z namespaces/openshift-service-ca-operator/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245541274Z namespaces/openshift-service-ca-operator/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245598215Z namespaces/openshift-service-ca-operator/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245637696Z namespaces/openshift-service-ca-operator/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245682207Z namespaces/openshift-service-ca-operator/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245721068Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24579754Z namespaces/openshift-service-ca-operator/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245879222Z namespaces/openshift-service-ca-operator/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245915713Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.245950234Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246003095Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246078317Z namespaces/openshift-service-ca-operator/monitoring.coreos.com/servicemonitors/service-ca-operator.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246131398Z namespaces/openshift-service-ca-operator/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246176719Z namespaces/openshift-service-ca-operator/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24621778Z namespaces/openshift-service-ca-operator/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24622398Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246262711Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246431355Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24664878Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.246789694Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2470513Z namespaces/openshift-service-ca-operator/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247263606Z namespaces/openshift-service-ca-operator/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247271716Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247325717Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/service-ca-operator-5d7b7dc95c-rfqh7.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24742843Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/service-ca-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24743859Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/service-ca-operator/service-ca-operator/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24744212Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/service-ca-operator/service-ca-operator/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247487081Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/service-ca-operator/service-ca-operator/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247638255Z namespaces/openshift-service-ca-operator/pods/service-ca-operator-5d7b7dc95c-rfqh7/service-ca-operator/service-ca-operator/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247779248Z namespaces/openshift-service-ca-operator/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247820819Z namespaces/openshift-service-ca-operator/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24785797Z namespaces/openshift-service-ca-operator/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247901601Z namespaces/openshift-service-ca-operator/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247946792Z namespaces/openshift-service-ca/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.247997873Z namespaces/openshift-service-ca/openshift-service-ca.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248086395Z namespaces/openshift-service-ca/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248122636Z namespaces/openshift-service-ca/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248173148Z namespaces/openshift-service-ca/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248210588Z namespaces/openshift-service-ca/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248292Z namespaces/openshift-service-ca/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248386853Z namespaces/openshift-service-ca/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248482445Z namespaces/openshift-service-ca/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248520506Z namespaces/openshift-service-ca/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248562177Z namespaces/openshift-service-ca/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248601938Z namespaces/openshift-service-ca/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248651059Z namespaces/openshift-service-ca/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248716731Z namespaces/openshift-service-ca/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248781532Z namespaces/openshift-service-ca/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248834844Z namespaces/openshift-service-ca/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248940766Z namespaces/openshift-service-ca/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248991937Z namespaces/openshift-service-ca/coordination.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.248999738Z namespaces/openshift-service-ca/coordination.k8s.io/leases/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249055209Z namespaces/openshift-service-ca/coordination.k8s.io/leases/service-ca-controller-lock.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24911005Z namespaces/openshift-service-ca/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249152991Z namespaces/openshift-service-ca/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249254224Z namespaces/openshift-service-ca/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249329116Z namespaces/openshift-service-ca/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249433208Z namespaces/openshift-service-ca/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24950756Z namespaces/openshift-service-ca/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249604172Z namespaces/openshift-service-ca/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249729105Z namespaces/openshift-service-ca/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249893429Z namespaces/openshift-service-ca/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.24994087Z namespaces/openshift-service-ca/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.249987472Z namespaces/openshift-service-ca/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250041043Z namespaces/openshift-service-ca/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250082754Z namespaces/openshift-service-ca/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250123795Z namespaces/openshift-service-ca/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250162326Z namespaces/openshift-service-ca/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250242478Z namespaces/openshift-service-ca/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25031308Z namespaces/openshift-service-ca/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25036127Z namespaces/openshift-service-ca/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250397671Z namespaces/openshift-service-ca/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250440792Z namespaces/openshift-service-ca/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250483713Z namespaces/openshift-service-ca/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250523825Z namespaces/openshift-service-ca/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250530245Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250567016Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25073877Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.250967425Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251220851Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251446817Z namespaces/openshift-service-ca/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251625311Z namespaces/openshift-service-ca/pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251634841Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251669772Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-7bb4f4bcc7-6bnkj.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251769975Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251780225Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-controller/service-ca-controller/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251791955Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-controller/service-ca-controller/logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.251803656Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-controller/service-ca-controller/logs/current.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252095753Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-controller/service-ca-controller/logs/previous.insecure.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252161594Z namespaces/openshift-service-ca/pods/service-ca-7bb4f4bcc7-6bnkj/service-ca-controller/service-ca-controller/logs/previous.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252194175Z namespaces/openshift-service-ca/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252239706Z namespaces/openshift-service-ca/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252275537Z namespaces/openshift-service-ca/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252319338Z namespaces/openshift-service-ca/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25237781Z namespaces/openshift-user-workload-monitoring/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25239716Z namespaces/openshift-user-workload-monitoring/openshift-user-workload-monitoring.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252462652Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252498682Z namespaces/openshift-user-workload-monitoring/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252535423Z namespaces/openshift-user-workload-monitoring/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252584384Z namespaces/openshift-user-workload-monitoring/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252656796Z namespaces/openshift-user-workload-monitoring/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252734368Z namespaces/openshift-user-workload-monitoring/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25280887Z namespaces/openshift-user-workload-monitoring/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252853561Z namespaces/openshift-user-workload-monitoring/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252886142Z namespaces/openshift-user-workload-monitoring/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252927873Z namespaces/openshift-user-workload-monitoring/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.252970394Z namespaces/openshift-user-workload-monitoring/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253082627Z namespaces/openshift-user-workload-monitoring/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253128588Z namespaces/openshift-user-workload-monitoring/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253164259Z namespaces/openshift-user-workload-monitoring/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253240611Z namespaces/openshift-user-workload-monitoring/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253289262Z namespaces/openshift-user-workload-monitoring/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253323322Z namespaces/openshift-user-workload-monitoring/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253420435Z namespaces/openshift-user-workload-monitoring/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253492437Z namespaces/openshift-user-workload-monitoring/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253570028Z namespaces/openshift-user-workload-monitoring/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25364072Z namespaces/openshift-user-workload-monitoring/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253725812Z namespaces/openshift-user-workload-monitoring/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.253840985Z namespaces/openshift-user-workload-monitoring/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254005189Z namespaces/openshift-user-workload-monitoring/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254098701Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254134992Z namespaces/openshift-user-workload-monitoring/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254182563Z namespaces/openshift-user-workload-monitoring/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254223724Z namespaces/openshift-user-workload-monitoring/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254265735Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254305596Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254378878Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25445471Z namespaces/openshift-user-workload-monitoring/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254490091Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254536572Z namespaces/openshift-user-workload-monitoring/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254579083Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254619494Z namespaces/openshift-user-workload-monitoring/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254658755Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254665135Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.254705286Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25486303Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255077825Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255220759Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255427964Z namespaces/openshift-user-workload-monitoring/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255610058Z namespaces/openshift-user-workload-monitoring/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255646289Z namespaces/openshift-user-workload-monitoring/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25569489Z namespaces/openshift-user-workload-monitoring/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255735341Z namespaces/openshift-user-workload-monitoring/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255772482Z namespaces/openshift/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255825743Z namespaces/openshift/openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255876954Z namespaces/openshift/apps.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255907805Z namespaces/openshift/apps.openshift.io/deploymentconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255956536Z namespaces/openshift/apps/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.255995137Z namespaces/openshift/apps/daemonsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256152331Z namespaces/openshift/apps/deployments.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256229163Z namespaces/openshift/apps/replicasets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256304205Z namespaces/openshift/apps/statefulsets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256407197Z namespaces/openshift/autoscaling/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256473539Z namespaces/openshift/autoscaling/horizontalpodautoscalers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25651927Z namespaces/openshift/batch/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256565781Z namespaces/openshift/batch/cronjobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256652113Z namespaces/openshift/batch/jobs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256691634Z namespaces/openshift/build.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256734605Z namespaces/openshift/build.openshift.io/buildconfigs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256806757Z namespaces/openshift/build.openshift.io/builds.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256859238Z namespaces/openshift/core/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256901319Z namespaces/openshift/core/configmaps.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.256996622Z namespaces/openshift/core/endpoints.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257101344Z namespaces/openshift/core/events.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257186046Z namespaces/openshift/core/persistentvolumeclaims.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257264518Z namespaces/openshift/core/pods.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25733731Z namespaces/openshift/core/replicationcontrollers.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257455893Z namespaces/openshift/core/secrets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257613847Z namespaces/openshift/core/services.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257658058Z namespaces/openshift/discovery.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257703479Z namespaces/openshift/discovery.k8s.io/endpointslices.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.25774377Z namespaces/openshift/image.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.257783601Z namespaces/openshift/image.openshift.io/imagestreams.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259050892Z namespaces/openshift/image.openshift.io/imagestreams/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259069132Z namespaces/openshift/image.openshift.io/imagestreams/cli-artifacts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259170134Z namespaces/openshift/image.openshift.io/imagestreams/cli.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259248196Z namespaces/openshift/image.openshift.io/imagestreams/dotnet-runtime.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259349789Z namespaces/openshift/image.openshift.io/imagestreams/dotnet.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259458541Z namespaces/openshift/image.openshift.io/imagestreams/driver-toolkit.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259541013Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift-java11.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259695737Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-eap-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259862461Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.259984974Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-java11-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260158838Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift-jdk11.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260256801Z namespaces/openshift/image.openshift.io/imagestreams/fuse7-karaf-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260370633Z namespaces/openshift/image.openshift.io/imagestreams/golang.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260482666Z namespaces/openshift/image.openshift.io/imagestreams/httpd.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260590939Z namespaces/openshift/image.openshift.io/imagestreams/installer-artifacts.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260670961Z namespaces/openshift/image.openshift.io/imagestreams/installer.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260762283Z namespaces/openshift/image.openshift.io/imagestreams/java-runtime.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260894606Z namespaces/openshift/image.openshift.io/imagestreams/java.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.260962908Z namespaces/openshift/image.openshift.io/imagestreams/jboss-datagrid73-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261097311Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261195273Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp3-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261288956Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261379858Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap-xp4-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26147146Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261566373Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk11-runtime-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261662555Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261748217Z namespaces/openshift/image.openshift.io/imagestreams/jboss-eap74-openjdk8-runtime-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26185232Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk11-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.261955862Z namespaces/openshift/image.openshift.io/imagestreams/jboss-webserver57-openjdk8-tomcat9-openshift-ubi8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262072285Z namespaces/openshift/image.openshift.io/imagestreams/jenkins-agent-base.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262170257Z namespaces/openshift/image.openshift.io/imagestreams/jenkins.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26227512Z namespaces/openshift/image.openshift.io/imagestreams/mariadb.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262387753Z namespaces/openshift/image.openshift.io/imagestreams/must-gather.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262470024Z namespaces/openshift/image.openshift.io/imagestreams/mysql.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262596018Z namespaces/openshift/image.openshift.io/imagestreams/network-tools.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262665939Z namespaces/openshift/image.openshift.io/imagestreams/nginx.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262785852Z namespaces/openshift/image.openshift.io/imagestreams/nodejs.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.262939836Z namespaces/openshift/image.openshift.io/imagestreams/oauth-proxy.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263021568Z namespaces/openshift/image.openshift.io/imagestreams/openjdk-11-rhel7.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263158631Z namespaces/openshift/image.openshift.io/imagestreams/perl.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263259943Z namespaces/openshift/image.openshift.io/imagestreams/php.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263370086Z namespaces/openshift/image.openshift.io/imagestreams/postgresql.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26352755Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso75-openshift-rhel8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263631153Z namespaces/openshift/image.openshift.io/imagestreams/postgresql13-for-sso76-openshift-rhel8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263736795Z namespaces/openshift/image.openshift.io/imagestreams/python.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.263884939Z namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264064733Z namespaces/openshift/image.openshift.io/imagestreams/redis.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264204426Z namespaces/openshift/image.openshift.io/imagestreams/ruby.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26433203Z namespaces/openshift/image.openshift.io/imagestreams/sso75-openshift-rhel8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264428102Z namespaces/openshift/image.openshift.io/imagestreams/sso76-openshift-rhel8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264515094Z namespaces/openshift/image.openshift.io/imagestreams/tests.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264610266Z namespaces/openshift/image.openshift.io/imagestreams/tools.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264721249Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11-runtime.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264836462Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-11.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.264953915Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17-runtime.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265092428Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-17.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265219131Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21-runtime.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265309023Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-21.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265401395Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8-runtime.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265516328Z namespaces/openshift/image.openshift.io/imagestreams/ubi8-openjdk-8.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26559516Z namespaces/openshift/k8s.ovn.org/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265634371Z namespaces/openshift/k8s.ovn.org/egressfirewalls.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265716113Z namespaces/openshift/k8s.ovn.org/egressqoses.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265794905Z namespaces/openshift/k8s.ovn.org/userdefinednetworks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265832386Z namespaces/openshift/monitoring.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265872797Z namespaces/openshift/monitoring.coreos.com/servicemonitors.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.265932268Z namespaces/openshift/networking.k8s.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26599707Z namespaces/openshift/networking.k8s.io/networkpolicies.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266069012Z namespaces/openshift/operators.coreos.com/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266088612Z namespaces/openshift/operators.coreos.com/clusterserviceversions/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266108193Z namespaces/openshift/operators.coreos.com/clusterserviceversions/authorino-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266256966Z namespaces/openshift/operators.coreos.com/clusterserviceversions/cert-manager-operator.v1.19.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26643Z namespaces/openshift/operators.coreos.com/clusterserviceversions/dns-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266561204Z namespaces/openshift/operators.coreos.com/clusterserviceversions/limitador-operator.v1.4.0.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266755138Z namespaces/openshift/operators.coreos.com/clusterserviceversions/rhcl-operator.v1.4.1.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.266964004Z namespaces/openshift/policy/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267003164Z namespaces/openshift/policy/poddisruptionbudgets.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267101767Z namespaces/openshift/route.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267136648Z namespaces/openshift/route.openshift.io/routes.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267186599Z namespaces/openshift/template.openshift.io/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267193199Z namespaces/openshift/template.openshift.io/templates/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26722906Z namespaces/openshift/template.openshift.io/templates/cache-service.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267338142Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267455455Z namespaces/openshift/template.openshift.io/templates/cakephp-mysql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267561668Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267690451Z namespaces/openshift/template.openshift.io/templates/dancer-mysql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267871426Z namespaces/openshift/template.openshift.io/templates/datagrid-service.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.267973138Z namespaces/openshift/template.openshift.io/templates/django-psql-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268137542Z namespaces/openshift/template.openshift.io/templates/django-psql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268248775Z namespaces/openshift/template.openshift.io/templates/eap-xp3-basic-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268370808Z namespaces/openshift/template.openshift.io/templates/eap-xp4-basic-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.26847883Z namespaces/openshift/template.openshift.io/templates/eap74-basic-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268588053Z namespaces/openshift/template.openshift.io/templates/eap74-https-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268713126Z namespaces/openshift/template.openshift.io/templates/eap74-sso-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268843619Z namespaces/openshift/template.openshift.io/templates/httpd-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.268942012Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral-monitored.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269144886Z namespaces/openshift/template.openshift.io/templates/jenkins-ephemeral.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269248059Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent-monitored.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269357502Z namespaces/openshift/template.openshift.io/templates/jenkins-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269463764Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269563527Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk11-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269667839Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-basic-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269764432Z namespaces/openshift/template.openshift.io/templates/jws57-openjdk8-tomcat9-ubi8-https-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269865904Z namespaces/openshift/template.openshift.io/templates/mariadb-ephemeral.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.269971206Z namespaces/openshift/template.openshift.io/templates/mariadb-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270097049Z namespaces/openshift/template.openshift.io/templates/mysql-ephemeral.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270197232Z namespaces/openshift/template.openshift.io/templates/mysql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270294784Z namespaces/openshift/template.openshift.io/templates/nginx-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270391877Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270502319Z namespaces/openshift/template.openshift.io/templates/nodejs-postgresql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270609282Z namespaces/openshift/template.openshift.io/templates/openjdk-web-basic-s2i.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270712214Z namespaces/openshift/template.openshift.io/templates/postgresql-ephemeral.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.270811357Z namespaces/openshift/template.openshift.io/templates/postgresql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.27092092Z namespaces/openshift/template.openshift.io/templates/rails-pgsql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271066303Z namespaces/openshift/template.openshift.io/templates/rails-postgresql-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271186926Z namespaces/openshift/template.openshift.io/templates/react-web-app-example.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271285388Z namespaces/openshift/template.openshift.io/templates/redis-ephemeral.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271382871Z namespaces/openshift/template.openshift.io/templates/redis-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271476553Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-rest-3scale.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271585845Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel-xml.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271686068Z namespaces/openshift/template.openshift.io/templates/s2i-fuse712-spring-boot-2-camel.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271798711Z namespaces/openshift/template.openshift.io/templates/sso75-https.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.271911333Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-https.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272021406Z namespaces/openshift/template.openshift.io/templates/sso75-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.27217971Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272299113Z namespaces/openshift/template.openshift.io/templates/sso75-postgresql.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272419276Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-https.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272563059Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272692312Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-postgresql.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272815355Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-https.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.272924338Z namespaces/openshift/template.openshift.io/templates/sso76-ocp4-x509-postgresql-persistent.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.27300448Z network_logs/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.273070161Z network_logs/cluster_scale [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.273154884Z network_logs/ippools.whereabouts.cni.cncf.io [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.273230196Z network_logs/multi-networkpolicy [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.273298547Z network_logs/net-attach-def [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.27340662Z network_logs/overlappingrangeipreservations.whereabouts.cni.cncf.io [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.273550903Z network_logs/ovn_kubernetes_top_pods [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.273644195Z network_logs/ovnk_database_store.tar.gz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.276127086Z nodes/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.276163767Z nodes/debug [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.276252439Z nodes/ip-10-0-132-117.ec2.internal/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.27629494Z nodes/ip-10-0-132-117.ec2.internal/cpu_affinities.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.276623818Z nodes/ip-10-0-132-117.ec2.internal/dmesg [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.276892325Z nodes/ip-10-0-132-117.ec2.internal/ethtool_channels [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.276964356Z nodes/ip-10-0-132-117.ec2.internal/ethtool_features [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.277072399Z nodes/ip-10-0-132-117.ec2.internal/ip-10-0-132-117.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.277517309Z nodes/ip-10-0-132-117.ec2.internal/irq_affinities.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.277672653Z nodes/ip-10-0-132-117.ec2.internal/lscpu [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.277799276Z nodes/ip-10-0-132-117.ec2.internal/lspci [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.277890658Z nodes/ip-10-0-132-117.ec2.internal/podresources.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.277990121Z nodes/ip-10-0-132-117.ec2.internal/pods_info.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.278142205Z nodes/ip-10-0-132-117.ec2.internal/proc_cmdline [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.278221477Z nodes/ip-10-0-132-117.ec2.internal/sysinfo.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.280449341Z nodes/ip-10-0-132-117.ec2.internal/sysinfo.tgz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.280607055Z nodes/ip-10-0-138-219.ec2.internal/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.280638935Z nodes/ip-10-0-138-219.ec2.internal/cpu_affinities.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.280915262Z nodes/ip-10-0-138-219.ec2.internal/dmesg [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.281215079Z nodes/ip-10-0-138-219.ec2.internal/ethtool_channels [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.281293321Z nodes/ip-10-0-138-219.ec2.internal/ethtool_features [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.281374583Z nodes/ip-10-0-138-219.ec2.internal/ip-10-0-138-219.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.281903856Z nodes/ip-10-0-138-219.ec2.internal/irq_affinities.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.282055Z nodes/ip-10-0-138-219.ec2.internal/lscpu [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.282191903Z nodes/ip-10-0-138-219.ec2.internal/lspci [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.282287475Z nodes/ip-10-0-138-219.ec2.internal/podresources.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.282388008Z nodes/ip-10-0-138-219.ec2.internal/pods_info.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.28249447Z nodes/ip-10-0-138-219.ec2.internal/proc_cmdline [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.282574342Z nodes/ip-10-0-138-219.ec2.internal/sysinfo.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.284367756Z nodes/ip-10-0-138-219.ec2.internal/sysinfo.tgz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.28451957Z nodes/ip-10-0-140-104.ec2.internal/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.284564411Z nodes/ip-10-0-140-104.ec2.internal/cpu_affinities.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.284854358Z nodes/ip-10-0-140-104.ec2.internal/dmesg [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.285162235Z nodes/ip-10-0-140-104.ec2.internal/ethtool_channels [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.285235897Z nodes/ip-10-0-140-104.ec2.internal/ethtool_features [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.285320679Z nodes/ip-10-0-140-104.ec2.internal/ip-10-0-140-104.ec2.internal_logs_kubelet.gz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.285656587Z nodes/ip-10-0-140-104.ec2.internal/irq_affinities.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2857929Z nodes/ip-10-0-140-104.ec2.internal/lscpu [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.285915813Z nodes/ip-10-0-140-104.ec2.internal/lspci [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.285999905Z nodes/ip-10-0-140-104.ec2.internal/podresources.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.286126208Z nodes/ip-10-0-140-104.ec2.internal/pods_info.json [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.286238041Z nodes/ip-10-0-140-104.ec2.internal/proc_cmdline [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.286312383Z nodes/ip-10-0-140-104.ec2.internal/sysinfo.log [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.288041765Z nodes/ip-10-0-140-104.ec2.internal/sysinfo.tgz [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.288190359Z pod_network_connectivity_check/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.2882328Z pod_network_connectivity_check/podnetworkconnectivitychecks.yaml [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.288330382Z static-pods/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.288337122Z static-pods/kube-apiserver/ [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.296874409Z [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.29689005Z sent 71,358 bytes received 10,104,433 bytes 6,783,860.67 bytes/sec [must-gather] [must-gather-h9gxr] OUT 2026-07-01T22:34:24.29689521Z total size is 117,084,292 speedup is 11.51 [must-gather] [must-gather ] OUT 2026-07-01T22:34:24.583366691Z namespace/openshift-must-gather-z7v9q deleted [must-gather] [must-gather] [must-gather] Reprinting Cluster State: [must-gather] When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: [must-gather] ClusterID: dad42e20-1340-4ed9-80d3-35f48ac16f25 [must-gather] ClientVersion: 4.21.10 [must-gather] ClusterVersion: Stable at "4.21.22" [must-gather] ClusterOperators: [must-gather] clusteroperator/authentication is missing [must-gather] clusteroperator/cloud-credential is missing [must-gather] clusteroperator/cluster-autoscaler is missing [must-gather] clusteroperator/config-operator is missing [must-gather] clusteroperator/etcd is missing [must-gather] clusteroperator/machine-api is missing [must-gather] clusteroperator/machine-approver is missing [must-gather] clusteroperator/machine-config is missing [must-gather] clusteroperator/marketplace is missing [must-gather] [must-gather] [git-push-artifacts] WORK_DIR: /workspace/odh-ci-artifacts [git-push-artifacts] REPO_PATH: opendatahub-io/odh-build-metadata [git-push-artifacts] REPO_BRANCH: ci-artifacts [git-push-artifacts] SPARSE_FILE_PATH: test-artifacts/docs [git-push-artifacts] SOURCE_PATH: /workspace/artifacts-dir [git-push-artifacts] DEST_PATH: test-artifacts/kserve-group-test-mpdk5 [git-push-artifacts] ALWAYS_PASS: false [git-push-artifacts] configuring gh token [git-push-artifacts] taking github token from Konflux bot [git-push-artifacts] Initialized empty Git repository in /workspace/odh-ci-artifacts/.git/ [git-push-artifacts] Using partial fetch with sparse checkout for: test-artifacts/docs [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] * [new branch] ci-artifacts -> origin/ci-artifacts [git-push-artifacts] Already on 'ci-artifacts' [git-push-artifacts] branch 'ci-artifacts' set up to track 'origin/ci-artifacts'. [git-push-artifacts] TASK_NAME=kserve-group-test-mpdk5-e2e-llm-inference-service [git-push-artifacts] PIPELINERUN_NAME=kserve-group-test-mpdk5 [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] -rw-r--r--. 1 root 1001540000 17897327 Jul 1 22:36 /workspace/odh-ci-artifacts/test-artifacts/kserve-group-test-mpdk5/e2e-llm-inference-service.tar.gz [git-push-artifacts] [ci-artifacts 3bf4d66] Updating CI Artifacts in e2e-llm-inference-service [git-push-artifacts] 1 file changed, 0 insertions(+), 0 deletions(-) [git-push-artifacts] create mode 100644 test-artifacts/kserve-group-test-mpdk5/e2e-llm-inference-service.tar.gz [git-push-artifacts] From https://github.com/opendatahub-io/odh-build-metadata [git-push-artifacts] * branch ci-artifacts -> FETCH_HEAD [git-push-artifacts] Already up to date. [git-push-artifacts] To https://github.com/opendatahub-io/odh-build-metadata.git [git-push-artifacts] 05d740f..3bf4d66 ci-artifacts -> ci-artifacts [fail-if-needed] Failing pipeline because deploy-and-e2e step failed container step-fail-if-needed has failed : [{"key":"StartedAt","value":"2026-07-01T22:36:08.536Z","type":3}]