: DataScienceCluster.datasciencecluster.opendatahub.io "default-dsc" is invalid: spec.components.modelregistry: Invalid value: "object": RegistriesNamespace is immutable when model registry is Managed { ErrStatus: { TypeMeta: {Kind: "Status", APIVersion: "v1"}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "DataScienceCluster.datasciencecluster.opendatahub.io \"default-dsc\" is invalid: spec.components.modelregistry: Invalid value: \"object\": RegistriesNamespace is immutable when model registry is Managed", Reason: "Invalid", Details: { Name: "default-dsc", Group: "datasciencecluster.opendatahub.io", Kind: "DataScienceCluster", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: \"object\": RegistriesNamespace is immutable when model registry is Managed", Field: "spec.components.modelregistry", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } occurred testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test 2026/09/15 17:43:07 TEST FAILURE DETECTED: group 2 2026/09/15 17:43:07 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/09/15 17:43:08 === CLUSTER STATE === 2026/09/15 17:43:08 Failed to collect node data: unhealthy nodes: ip-10-0-1-39.us-east-2.compute.internal (Ready=False) 2026/09/15 17:43:08 === DEPLOYMENTS === 2026/09/15 17:43:08 Namespace: redhat-ods-applications 2026/09/15 17:43:08 Deployment aihub-controller-manager: 1/1 (READY) 2026/09/15 17:43:08 Deployment catalog-controller-manager: 1/1 (READY) 2026/09/15 17:43:08 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/09/15 17:43:08 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/09/15 17:43:08 Deployment odh-observability: 1/1 (READY) 2026/09/15 17:43:08 Deployment spark-operator-controller: 1/1 (READY) 2026/09/15 17:43:08 Deployment spark-operator-module-controller-manager: 1/1 (READY) 2026/09/15 17:43:08 Deployment spark-operator-webhook: 1/1 (READY) 2026/09/15 17:43:08 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/09/15 17:43:08 Namespace: kube-system 2026/09/15 17:43:08 === PODS === 2026/09/15 17:43:08 Failed to collect pod data: kube-system/global-pull-secret-syncer-6hk7w: phase=Pending 2026/09/15 17:43:08 Continuing with partial pod data to collect logs from problematic containers 2026/09/15 17:43:08 Namespace redhat-ods-applications: no problematic pods found 2026/09/15 17:43:08 Namespace: kube-system 2026/09/15 17:43:08 Pod global-pull-secret-syncer-6hk7w: Pending 2026/09/15 17:43:08 Container global-pull-secret-syncer: ready=false, waiting: ContainerCreating 2026/09/15 17:43:08 === OPERATOR STATUS === 2026/09/15 17:43:08 Operator deployment: 3/3 ready 2026/09/15 17:43:08 === DSCI STATUS === 2026/09/15 17:43:08 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; ThanosQuerierAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; NodeMetricsEndpointAvailable=False; PersesAvailable=False 2026/09/15 17:43:08 === DSC STATUS === 2026/09/15 17:43:08 DSC default-dsc: 2026/09/15 17:43:08 Ready: True - 2026/09/15 17:43:08 ProvisioningSucceeded: True - 2026/09/15 17:43:08 ComponentsReady: True - 2026/09/15 17:43:08 ModulesReady: True - 2026/09/15 17:43:08 AIGatewayReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/09/15 17:43:08 BatchGatewayReady: False - Submodule ManagementState is set to Removed 2026/09/15 17:43:08 DashboardReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 FeastOperatorReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 KserveLLMInferenceServiceDependencies: False - Submodule ManagementState is set to Removed 2026/09/15 17:43:08 KserveLLMInferenceServiceWideEPDependencies: False - Submodule ManagementState is set to Removed 2026/09/15 17:43:08 KserveReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 KueueReady: False - Component ManagementState is set to Removed 2026/09/15 17:43:08 MCPLifecycleOperatorReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 MLflowOperatorReady: True - MLflowOperator is ready to manage MLflow custom resources 2026/09/15 17:43:08 ModelRegistryReady: True - 2026/09/15 17:43:08 ModelsAsAServiceReady: False - Submodule ManagementState is set to Removed 2026/09/15 17:43:08 OGXReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 RayReady: False - Component ManagementState is set to Removed 2026/09/15 17:43:08 SparkOperatorReady: True - 2026/09/15 17:43:08 TrainerReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 TrustyAIReady: True - 2026/09/15 17:43:08 WorkbenchesReady: False - Module ManagementState is set to Removed 2026/09/15 17:43:08 WorkbenchesV2Ready: False - Submodule ManagementState is set to Removed 2026/09/15 17:43:08 === RECENT EVENTS === 2026/09/15 17:43:08 17:40:17 WARN Pod/global-pull-secret-syncer-6hk7w: NetworkNotReady - network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? 2026/09/15 17:43:08 17:40:08 WARN Pod/global-pull-secret-syncer-6hk7w: FailedMount - MountVolume.SetUp failed for volume "global-pull-secret" : object "kube-system"/"global-pull-secret" not registered 2026/09/15 17:43:08 17:40:08 WARN Pod/global-pull-secret-syncer-6hk7w: FailedMount - MountVolume.SetUp failed for volume "original-pull-secret" : object "kube-system"/"original-pull-secret" not registered 2026/09/15 17:43:08 17:40:03 INFO Pod/konnectivity-agent-g2dn4: Created - Created container: konnectivity-agent 2026/09/15 17:43:08 17:40:03 INFO Pod/konnectivity-agent-g2dn4: Started - Started container konnectivity-agent 2026/09/15 17:43:08 17:40:02 INFO Pod/kube-apiserver-proxy-ip-10-0-1-39.us-east-2.compute.internal: Started - Started container haproxy 2026/09/15 17:43:08 17:40:02 INFO Pod/kube-apiserver-proxy-ip-10-0-1-39.us-east-2.compute.internal: Pulled - Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:469457eef00af3ae395c1f8e4dcc354faa680360ba813a14bf2d260cf46887b4" in 9.612s (9.612s including waiting). Image size: 482987592 bytes. 2026/09/15 17:43:08 17:40:02 INFO Pod/kube-apiserver-proxy-ip-10-0-1-39.us-east-2.compute.internal: Created - Created container: haproxy 2026/09/15 17:43:08 17:40:02 INFO Pod/konnectivity-agent-g2dn4: Pulled - Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98880a999bdb21bf52b5b635d08fc59ccf26475da454e2306428e5da55db416b" in 8.856s (8.856s including waiting). Image size: 468939173 bytes. 2026/09/15 17:43:08 17:39:54 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:43:08 17:39:54 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:43:08 17:39:53 INFO Pod/konnectivity-agent-g2dn4: Pulling - Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98880a999bdb21bf52b5b635d08fc59ccf26475da454e2306428e5da55db416b" 2026/09/15 17:43:08 17:39:52 INFO Pod/global-pull-secret-syncer-6hk7w: Scheduled - Successfully assigned kube-system/global-pull-secret-syncer-6hk7w to ip-10-0-1-39.us-east-2.compute.internal 2026/09/15 17:43:08 17:39:52 INFO DaemonSet/global-pull-secret-syncer: SuccessfulCreate - Created pod: global-pull-secret-syncer-6hk7w 2026/09/15 17:43:08 17:39:52 INFO Pod/kube-apiserver-proxy-ip-10-0-1-39.us-east-2.compute.internal: Pulling - Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:469457eef00af3ae395c1f8e4dcc354faa680360ba813a14bf2d260cf46887b4" 2026/09/15 17:43:08 17:39:51 INFO Pod/konnectivity-agent-g2dn4: Scheduled - Successfully assigned kube-system/konnectivity-agent-g2dn4 to ip-10-0-1-39.us-east-2.compute.internal 2026/09/15 17:43:08 17:39:51 INFO DaemonSet/konnectivity-agent: SuccessfulCreate - Created pod: konnectivity-agent-g2dn4 2026/09/15 17:43:08 17:39:15 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:43:08 17:39:15 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:43:08 17:38:39 WARN Pod/rhods-dashboard-55c678f9b-2j4xc: Unhealthy - Readiness probe failed: Get "https://10.129.71.214:8943/healthcheck": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2026/09/15 17:43:08 17:38:30 WARN Pod/rhods-dashboard-55c678f9b-2j4xc: Unhealthy - Readiness probe failed: Get "https://10.129.71.214:8444/healthz": context deadline exceeded 2026/09/15 17:43:08 17:38:28 WARN Pod/rhods-dashboard-6bcf96c49-xqc5r: Unhealthy - Readiness probe failed: Get "https://10.130.21.229:8444/healthz": dial tcp 10.130.21.229:8444: connect: connection refused 2026/09/15 17:43:08 17:38:28 WARN Pod/rhods-dashboard-6bcf96c49-xqc5r: Unhealthy - Readiness probe failed: Get "https://10.130.21.229:8943/healthcheck": dial tcp 10.130.21.229:8943: connect: connection refused 2026/09/15 17:43:08 17:38:26 WARN Pod/rhods-dashboard-55c678f9b-2j4xc: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/15 17:43:08 17:38:25 WARN Pod/rhods-dashboard-6bcf96c49-xqc5r: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/15 17:43:08 17:38:24 WARN Pod/rhods-dashboard-55c678f9b-2j4xc: Unhealthy - Readiness probe failed: Get "https://10.129.71.214:8444/healthz": dial tcp 10.129.71.214:8444: connect: connection refused 2026/09/15 17:43:08 17:38:24 WARN Pod/rhods-dashboard-55c678f9b-2j4xc: Unhealthy - Readiness probe failed: Get "https://10.129.71.214:8943/healthcheck": dial tcp 10.129.71.214:8943: connect: connection refused 2026/09/15 17:43:08 17:38:22 WARN Pod/rhods-dashboard-6bcf96c49-8vd77: Unhealthy - Readiness probe failed: Get "https://10.129.71.213:8444/healthz": dial tcp 10.129.71.213:8444: connect: connection refused 2026/09/15 17:43:08 17:38:22 WARN Pod/rhods-dashboard-6bcf96c49-8vd77: Unhealthy - Readiness probe failed: Get "https://10.129.71.213:8943/healthcheck": dial tcp 10.129.71.213:8943: connect: connection refused 2026/09/15 17:43:08 17:38:19 WARN Pod/rhods-dashboard-6bcf96c49-8vd77: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/15 17:43:08 17:38:18 WARN Pod/rhods-dashboard-55c678f9b-jwhfx: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/15 17:43:08 17:38:16 WARN Pod/rhods-dashboard-55c678f9b-jwhfx: Unhealthy - Readiness probe failed: Get "https://10.130.21.230:8444/healthz": dial tcp 10.130.21.230:8444: connect: connection refused 2026/09/15 17:43:08 17:38:16 WARN Pod/rhods-dashboard-55c678f9b-jwhfx: Unhealthy - Readiness probe failed: Get "https://10.130.21.230:8943/healthcheck": dial tcp 10.130.21.230:8943: connect: connection refused 2026/09/15 17:43:08 17:38:15 WARN Pod/rhods-dashboard-55c678f9b-jwhfx: FailedMount - MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "kube-rbac-proxy-config" not found 2026/09/15 17:43:08 17:38:15 WARN Pod/rhods-dashboard-55c678f9b-jwhfx: FailedMount - MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 2026/09/15 17:43:08 17:38:12 WARN Pod/rhods-dashboard-55c678f9b-ss4sv: Unhealthy - Readiness probe failed: Get "https://10.130.21.220:8943/healthcheck": dial tcp 10.130.21.220:8943: connect: connection refused 2026/09/15 17:43:08 17:38:12 WARN Pod/rhods-dashboard-55c678f9b-ss4sv: Unhealthy - Readiness probe failed: Get "https://10.130.21.220:8444/healthz": dial tcp 10.130.21.220:8444: connect: connection refused 2026/09/15 17:43:08 17:38:11 WARN Pod/rhods-dashboard-55c678f9b-pcwv5: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/15 17:43:08 17:38:10 WARN Pod/rhods-dashboard-55c678f9b-ss4sv: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/15 17:43:08 17:38:08 WARN Pod/rhods-dashboard-55c678f9b-pcwv5: Unhealthy - Readiness probe failed: Get "https://10.129.71.202:8444/healthz": dial tcp 10.129.71.202:8444: connect: connection refused 2026/09/15 17:43:08 17:38:08 WARN Pod/rhods-dashboard-55c678f9b-pcwv5: Unhealthy - Readiness probe failed: Get "https://10.129.71.202:8943/healthcheck": dial tcp 10.129.71.202:8943: connect: connection refused 2026/09/15 17:43:08 === RESOURCE QUOTAS === 2026/09/15 17:43:08 No resource quotas found in monitored namespaces 2026/09/15 17:43:08 Classification for group 2: infrastructure/pod-startup (code=1002, confidence=high) 2026/09/15 17:43:08 Evidence: pod global-pull-secret-syncer-6hk7w stuck in Pending for 3m16s FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["pod global-pull-secret-syncer-6hk7w stuck in Pending for 3m16s"],"confidence":"high"} 2026/09/15 17:43:08 Diagnostics complete for failed test: group 2 controller_test.go:769: Stopping: group 2 test failed. 2026/09/15 17:43:08 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/09/15 17:43:08 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026-09-15T17:49:03Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.6.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.20.12"}} 2026/09/15 17:52:14 TEST FAILURE DETECTED: components 2026/09/15 17:52:14 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/09/15 17:52:15 === CLUSTER STATE === 2026/09/15 17:52:15 Node: ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 17:52:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 17:52:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 17:52:15 Condition Ready: True - kubelet is posting ready status 2026/09/15 17:52:15 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/09/15 17:52:15 Node: ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/15 17:52:15 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/15 17:52:15 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/15 17:52:15 Condition Ready: True - kubelet is posting ready status 2026/09/15 17:52:15 Resources: 7500m CPU, 31236232Ki memory allocatable (of 8000m CPU, 32387208Ki memory capacity) 2026/09/15 17:52:15 === DEPLOYMENTS === 2026/09/15 17:52:15 Namespace: redhat-ods-applications 2026/09/15 17:52:15 Deployment aihub-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment catalog-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment kserve-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment kserve-localmodel-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment kserve-module-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment model-serving-api: 1/1 (READY) 2026/09/15 17:52:15 Deployment odh-model-controller: 1/1 (READY) 2026/09/15 17:52:15 Deployment odh-observability: 1/1 (READY) 2026/09/15 17:52:15 Deployment spark-operator-controller: 1/1 (READY) 2026/09/15 17:52:15 Deployment spark-operator-module-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Deployment spark-operator-webhook: 1/1 (READY) 2026/09/15 17:52:15 Deployment trainer-operator-controller-manager: 1/1 (READY) 2026/09/15 17:52:15 Namespace: kube-system 2026/09/15 17:52:15 === PODS === 2026/09/15 17:52:15 Namespace redhat-ods-applications: no problematic pods found 2026/09/15 17:52:15 Namespace kube-system: no problematic pods found 2026/09/15 17:52:15 === OPERATOR STATUS === 2026/09/15 17:52:15 Operator deployment: 3/3 ready 2026/09/15 17:52:15 === DSCI STATUS === 2026/09/15 17:52:15 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; ThanosQuerierAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; NodeMetricsEndpointAvailable=False; PersesAvailable=False 2026/09/15 17:52:15 === DSC STATUS === 2026/09/15 17:52:15 DSC default-dsc: 2026/09/15 17:52:15 Ready: True - 2026/09/15 17:52:15 ProvisioningSucceeded: True - 2026/09/15 17:52:15 ComponentsReady: True - All registered components have ManagementState Removed or are not configured 2026/09/15 17:52:15 ModulesReady: True - 2026/09/15 17:52:15 AIGatewayReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/09/15 17:52:15 BatchGatewayReady: False - Submodule ManagementState is set to Removed 2026/09/15 17:52:15 DashboardReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 FeastOperatorReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 KserveLLMInferenceServiceDependencies: True - 2026/09/15 17:52:15 KserveLLMInferenceServiceWideEPDependencies: True - 2026/09/15 17:52:15 KserveReady: True - 2026/09/15 17:52:15 KueueReady: False - Component ManagementState is set to Removed 2026/09/15 17:52:15 MCPLifecycleOperatorReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 MLflowOperatorReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 ModelRegistryReady: True - 2026/09/15 17:52:15 ModelsAsAServiceReady: False - Submodule ManagementState is set to Removed 2026/09/15 17:52:15 OGXReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 RayReady: False - Component ManagementState is set to Removed 2026/09/15 17:52:15 SparkOperatorReady: True - 2026/09/15 17:52:15 TrainerReady: True - 2026/09/15 17:52:15 TrustyAIReady: False - Component ManagementState is set to Removed 2026/09/15 17:52:15 WorkbenchesReady: False - Module ManagementState is set to Removed 2026/09/15 17:52:15 WorkbenchesV2Ready: False - Submodule ManagementState is set to Removed 2026/09/15 17:52:15 === RECENT EVENTS === 2026/09/15 17:52:15 17:51:53 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7945ffcd-nskbg_150a5f8c-f31e-490e-93fe-c26333fa8676 became leader 2026/09/15 17:52:15 17:51:52 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-6dffbf4d96-9r2hq_3e3353f8-4a6e-4d9f-ab31-9c378bc88cd8 became leader 2026/09/15 17:52:15 17:51:52 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-8754f79ff-9nq9x_6e996437-7572-4263-9432-39dc91b913bb became leader 2026/09/15 17:52:15 17:51:41 INFO Pod/kserve-localmodelnode-agent-htzlv: Created - Created container: manager 2026/09/15 17:52:15 17:51:41 INFO Pod/kserve-localmodelnode-agent-htzlv: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 758ms (758ms including waiting). Image size: 245681483 bytes. 2026/09/15 17:52:15 17:51:41 INFO Pod/kserve-localmodelnode-agent-6zjvh: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 817ms (817ms including waiting). Image size: 245681483 bytes. 2026/09/15 17:52:15 17:51:41 INFO Pod/kserve-localmodelnode-agent-6zjvh: Created - Created container: manager 2026/09/15 17:52:15 17:51:41 INFO Pod/kserve-localmodelnode-agent-6zjvh: Started - Started container manager 2026/09/15 17:52:15 17:51:41 INFO Pod/kserve-localmodelnode-agent-htzlv: Started - Started container manager 2026/09/15 17:52:15 17:51:40 INFO Pod/kserve-localmodelnode-agent-6zjvh: AddedInterface - Add eth0 [10.129.71.238/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:40 INFO Pod/kserve-localmodelnode-agent-htzlv: AddedInterface - Add eth0 [10.130.21.247/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:40 INFO Pod/kserve-localmodelnode-agent-6zjvh: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/15 17:52:15 17:51:40 INFO Pod/kserve-localmodelnode-agent-htzlv: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/15 17:52:15 17:51:39 INFO Pod/kserve-localmodelnode-agent-htzlv: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-htzlv to ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:39 INFO Pod/kserve-localmodelnode-agent-6zjvh: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-6zjvh to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:36 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" in 790ms (790ms including waiting). Image size: 245939552 bytes. 2026/09/15 17:52:15 17:51:36 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: Created - Created container: manager 2026/09/15 17:52:15 17:51:36 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: Started - Started container manager 2026/09/15 17:52:15 17:51:35 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" 2026/09/15 17:52:15 17:51:35 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: AddedInterface - Add eth0 [10.129.71.236/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:35 INFO Pod/llmisvc-controller-manager-6dffbf4d96-9r2hq: Started - Started container manager 2026/09/15 17:52:15 17:51:35 INFO Pod/odh-model-controller-8754f79ff-9nq9x: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" in 869ms (869ms including waiting). Image size: 235603578 bytes. 2026/09/15 17:52:15 17:51:35 INFO Pod/odh-model-controller-8754f79ff-9nq9x: Created - Created container: manager 2026/09/15 17:52:15 17:51:35 INFO Pod/odh-model-controller-8754f79ff-9nq9x: Started - Started container manager 2026/09/15 17:52:15 17:51:34 INFO Pod/kserve-controller-manager-7945ffcd-nskbg: AddedInterface - Add eth0 [10.129.71.235/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:34 INFO Pod/model-serving-api-74db48b547-skcg4: Started - Started container server 2026/09/15 17:52:15 17:51:34 INFO Pod/llmisvc-controller-manager-6dffbf4d96-9r2hq: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" in 990ms (991ms including waiting). Image size: 235369934 bytes. 2026/09/15 17:52:15 17:51:34 INFO Pod/llmisvc-controller-manager-6dffbf4d96-9r2hq: Created - Created container: manager 2026/09/15 17:52:15 17:51:34 INFO Pod/odh-model-controller-8754f79ff-9nq9x: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" 2026/09/15 17:52:15 17:51:34 INFO Pod/model-serving-api-74db48b547-skcg4: AddedInterface - Add eth0 [10.130.21.245/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:34 INFO Pod/odh-model-controller-8754f79ff-9nq9x: AddedInterface - Add eth0 [10.129.71.237/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:34 WARN Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found 2026/09/15 17:52:15 17:51:34 INFO Pod/model-serving-api-74db48b547-skcg4: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:ad1c5ec4f15df0dff60ef6d93886e674b4bc512d20da31598b84ac5a009a13b3" already present on machine 2026/09/15 17:52:15 17:51:34 INFO Pod/kserve-controller-manager-7945ffcd-nskbg: Started - Started container manager 2026/09/15 17:52:15 17:51:34 INFO Pod/kserve-controller-manager-7945ffcd-nskbg: Created - Created container: manager 2026/09/15 17:52:15 17:51:34 INFO Pod/kserve-controller-manager-7945ffcd-nskbg: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" in 719ms (719ms including waiting). Image size: 247701913 bytes. 2026/09/15 17:52:15 17:51:34 INFO Pod/kserve-controller-manager-7945ffcd-nskbg: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" 2026/09/15 17:52:15 17:51:34 INFO Pod/model-serving-api-74db48b547-skcg4: Created - Created container: server 2026/09/15 17:52:15 17:51:33 WARN Pod/kserve-localmodelnode-agent-htzlv: FailedScheduling - 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/09/15 17:52:15 17:51:33 WARN Pod/kserve-localmodelnode-agent-6zjvh: FailedScheduling - 0/2 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. 2026/09/15 17:52:15 17:51:33 INFO Pod/odh-model-controller-8754f79ff-9nq9x: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-8754f79ff-9nq9x to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:33 INFO Pod/llmisvc-controller-manager-6dffbf4d96-9r2hq: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-6dffbf4d96-9r2hq to ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:33 INFO Pod/model-serving-api-74db48b547-skcg4: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-74db48b547-skcg4 to ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:33 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodel-controller-manager-76664cd8c7-l2dnl to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:33 INFO Pod/kserve-controller-manager-7945ffcd-nskbg: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7945ffcd-nskbg to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:51:33 INFO ReplicaSet/model-serving-api-74db48b547: SuccessfulCreate - Created pod: model-serving-api-74db48b547-skcg4 2026/09/15 17:52:15 17:51:33 INFO ReplicaSet/odh-model-controller-8754f79ff: SuccessfulCreate - Created pod: odh-model-controller-8754f79ff-9nq9x 2026/09/15 17:52:15 17:51:33 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-6dffbf4d96 from 0 to 1 2026/09/15 17:52:15 17:51:33 INFO ReplicaSet/kserve-localmodel-controller-manager-76664cd8c7: SuccessfulCreate - Created pod: kserve-localmodel-controller-manager-76664cd8c7-l2dnl 2026/09/15 17:52:15 17:51:33 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:51:33 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7945ffcd from 0 to 1 2026/09/15 17:52:15 17:51:33 INFO Pod/llmisvc-controller-manager-6dffbf4d96-9r2hq: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" 2026/09/15 17:52:15 17:51:33 INFO Pod/llmisvc-controller-manager-6dffbf4d96-9r2hq: AddedInterface - Add eth0 [10.130.21.246/23] from ovn-kubernetes 2026/09/15 17:52:15 17:51:33 INFO ReplicaSet/kserve-controller-manager-7945ffcd: SuccessfulCreate - Created pod: kserve-controller-manager-7945ffcd-nskbg 2026/09/15 17:52:15 17:51:33 WARN Pod/model-serving-api-74db48b547-skcg4: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/09/15 17:52:15 17:51:33 INFO ReplicaSet/llmisvc-controller-manager-6dffbf4d96: SuccessfulCreate - Created pod: llmisvc-controller-manager-6dffbf4d96-9r2hq 2026/09/15 17:52:15 17:51:33 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-8754f79ff from 0 to 1 2026/09/15 17:52:15 17:51:33 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:51:33 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "openshift-ai-llminferenceservice-multi-node-scc": Forbidden: not usable by user or serviceaccount, provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1001190000, 1001199999], provider "restricted-v3": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "pcap-dedicated-admi ns": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/15 17:52:15 17:51:33 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-6zjvh 2026/09/15 17:52:15 17:51:33 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-74db48b547 from 0 to 1 2026/09/15 17:52:15 17:51:33 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-htzlv 2026/09/15 17:52:15 17:51:33 WARN Pod/odh-model-controller-8754f79ff-9nq9x: FailedMount - MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 2026/09/15 17:52:15 17:51:33 INFO Deployment/kserve-localmodel-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-localmodel-controller-manager-76664cd8c7 from 0 to 1 2026/09/15 17:52:15 17:51:33 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "openshift-ai-llminferenceservice-multi-node-scc": Forbidden: not usable by user or serviceaccount, provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1001190000, 1001199999], provider "restricted-v3": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "pcap-dedicated-admi ns": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/15 17:52:15 17:51:07 INFO Lease/kserve-module-controller-manager: LeaderElection - kserve-module-controller-manager-858fdb6fd-bhszp_b1e31059-1c33-4531-b227-75719b5c41a7 became leader 2026/09/15 17:52:15 17:50:46 INFO Pod/kserve-module-controller-manager-858fdb6fd-bhszp: Started - Started container manager 2026/09/15 17:52:15 17:50:46 INFO Pod/kserve-module-controller-manager-858fdb6fd-bhszp: Created - Created container: manager 2026/09/15 17:52:15 17:50:46 INFO Pod/kserve-module-controller-manager-858fdb6fd-bhszp: Pulled - Container image "quay.io/rhoai/odh-kserve-module-operator-rhel9@sha256:7caa7a001cc41b2cdb15f529cde6e0a3b3bc8784e959fb935716f9d312485abc" already present on machine 2026/09/15 17:52:15 17:50:46 INFO Pod/kserve-module-controller-manager-858fdb6fd-bhszp: AddedInterface - Add eth0 [10.129.71.234/23] from ovn-kubernetes 2026/09/15 17:52:15 17:50:45 INFO Pod/kserve-module-controller-manager-858fdb6fd-bhszp: Scheduled - Successfully assigned redhat-ods-applications/kserve-module-controller-manager-858fdb6fd-bhszp to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:50:45 INFO ReplicaSet/kserve-module-controller-manager-858fdb6fd: SuccessfulCreate - Created pod: kserve-module-controller-manager-858fdb6fd-bhszp 2026/09/15 17:52:15 17:50:45 INFO Deployment/kserve-module-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-module-controller-manager-858fdb6fd from 0 to 1 2026/09/15 17:52:15 17:50:38 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:38 INFO Pod/odh-model-controller-8754f79ff-dl5wp: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:38 INFO Pod/model-serving-api-74db48b547-8x2qr: Killing - Stopping container server 2026/09/15 17:52:15 17:50:38 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:38 INFO Pod/kserve-module-controller-manager-858fdb6fd-g68rz: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:38 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:34 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:50:34 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-8t82m 2026/09/15 17:52:15 17:50:34 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-zzdz5 2026/09/15 17:52:15 17:50:34 INFO Pod/kserve-localmodelnode-agent-zzdz5: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:34 INFO Pod/kserve-localmodelnode-agent-8t82m: Killing - Stopping container manager 2026/09/15 17:52:15 17:50:13 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7945ffcd-fkzsg_588d7d84-2929-4992-83a6-023099834db1 became leader 2026/09/15 17:52:15 17:50:13 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-6dffbf4d96-x7qj8_b0699dcd-e093-4c08-915d-f0a4d343093b became leader 2026/09/15 17:52:15 17:50:12 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-8754f79ff-dl5wp_7a13065e-bdff-4d89-aef6-dd0bbb89ddbf became leader 2026/09/15 17:52:15 17:49:55 INFO Pod/odh-model-controller-8754f79ff-dl5wp: Created - Created container: manager 2026/09/15 17:52:15 17:49:55 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: Started - Started container manager 2026/09/15 17:52:15 17:49:55 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: Created - Created container: manager 2026/09/15 17:52:15 17:49:55 INFO Pod/odh-model-controller-8754f79ff-dl5wp: Started - Started container manager 2026/09/15 17:52:15 17:49:55 INFO Pod/odh-model-controller-8754f79ff-dl5wp: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" in 783ms (783ms including waiting). Image size: 235603578 bytes. 2026/09/15 17:52:15 17:49:55 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" in 831ms (831ms including waiting). Image size: 247701913 bytes. 2026/09/15 17:52:15 17:49:54 INFO Pod/odh-model-controller-8754f79ff-dl5wp: AddedInterface - Add eth0 [10.130.21.242/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: Started - Started container manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: AddedInterface - Add eth0 [10.129.71.230/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:54 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" in 945ms (945ms including waiting). Image size: 235369934 bytes. 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" in 876ms (876ms including waiting). Image size: 245939552 bytes. 2026/09/15 17:52:15 17:49:54 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: Created - Created container: manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: Created - Created container: manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" 2026/09/15 17:52:15 17:49:54 INFO Pod/odh-model-controller-8754f79ff-dl5wp: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" 2026/09/15 17:52:15 17:49:54 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: Started - Started container manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodelnode-agent-zzdz5: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 861ms (861ms including waiting). Image size: 245681483 bytes. 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodelnode-agent-8t82m: Started - Started container manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodelnode-agent-8t82m: Created - Created container: manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodelnode-agent-zzdz5: Created - Created container: manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodelnode-agent-zzdz5: Started - Started container manager 2026/09/15 17:52:15 17:49:54 INFO Pod/kserve-localmodelnode-agent-8t82m: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 861ms (861ms including waiting). Image size: 245681483 bytes. 2026/09/15 17:52:15 17:49:53 INFO Pod/kserve-localmodelnode-agent-zzdz5: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/15 17:52:15 17:49:53 INFO Pod/model-serving-api-74db48b547-8x2qr: Started - Started container server 2026/09/15 17:52:15 17:49:53 INFO Pod/model-serving-api-74db48b547-8x2qr: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:ad1c5ec4f15df0dff60ef6d93886e674b4bc512d20da31598b84ac5a009a13b3" already present on machine 2026/09/15 17:52:15 17:49:53 INFO Pod/model-serving-api-74db48b547-8x2qr: Created - Created container: server 2026/09/15 17:52:15 17:49:53 INFO Pod/kserve-localmodelnode-agent-8t82m: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/15 17:52:15 17:49:53 INFO Pod/kserve-localmodelnode-agent-8t82m: AddedInterface - Add eth0 [10.129.71.233/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:53 INFO Pod/model-serving-api-74db48b547-8x2qr: AddedInterface - Add eth0 [10.130.21.241/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:53 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: AddedInterface - Add eth0 [10.129.71.231/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:53 INFO Pod/kserve-localmodelnode-agent-zzdz5: AddedInterface - Add eth0 [10.130.21.243/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:53 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" 2026/09/15 17:52:15 17:49:53 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: AddedInterface - Add eth0 [10.129.71.232/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:53 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" 2026/09/15 17:52:15 17:49:53 WARN Pod/kserve-controller-manager-7945ffcd-fkzsg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/09/15 17:52:15 17:49:53 WARN Pod/odh-model-controller-8754f79ff-dl5wp: FailedMount - MountVolume.SetUp failed for volume "metrics-certs" : secret "odh-model-controller-metrics-tls" not found 2026/09/15 17:52:15 17:49:52 INFO Pod/kserve-localmodelnode-agent-zzdz5: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-zzdz5 to ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO Pod/kserve-localmodelnode-agent-8t82m: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-8t82m to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO Pod/odh-model-controller-8754f79ff-dl5wp: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-8754f79ff-dl5wp to ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO Pod/model-serving-api-74db48b547-8x2qr: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-74db48b547-8x2qr to ip-10-0-1-61.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO Pod/llmisvc-controller-manager-6dffbf4d96-x7qj8: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-6dffbf4d96-x7qj8 to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-7x6nl: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodel-controller-manager-76664cd8c7-7x6nl to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO Pod/kserve-controller-manager-7945ffcd-fkzsg: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7945ffcd-fkzsg to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:52 INFO ReplicaSet/model-serving-api-74db48b547: SuccessfulCreate - Created pod: model-serving-api-74db48b547-8x2qr 2026/09/15 17:52:15 17:49:52 INFO ReplicaSet/odh-model-controller-8754f79ff: SuccessfulCreate - Created pod: odh-model-controller-8754f79ff-dl5wp 2026/09/15 17:52:15 17:49:52 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-zzdz5 2026/09/15 17:52:15 17:49:52 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7945ffcd from 0 to 1 2026/09/15 17:52:15 17:49:52 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:49:52 INFO ReplicaSet/kserve-localmodel-controller-manager-76664cd8c7: SuccessfulCreate - Created pod: kserve-localmodel-controller-manager-76664cd8c7-7x6nl 2026/09/15 17:52:15 17:49:52 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-8754f79ff from 0 to 1 2026/09/15 17:52:15 17:49:52 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-8t82m 2026/09/15 17:52:15 17:49:52 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "openshift-ai-llminferenceservice-multi-node-scc": Forbidden: not usable by user or serviceaccount, provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1001190000, 1001199999], provider "restricted-v3": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "pcap-dedicated-admi ns": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/15 17:52:15 17:49:52 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-74db48b547 from 0 to 1 2026/09/15 17:52:15 17:49:52 INFO ReplicaSet/kserve-controller-manager-7945ffcd: SuccessfulCreate - Created pod: kserve-controller-manager-7945ffcd-fkzsg 2026/09/15 17:52:15 17:49:52 INFO Deployment/kserve-localmodel-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-localmodel-controller-manager-76664cd8c7 from 0 to 1 2026/09/15 17:52:15 17:49:52 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "openshift-ai-llminferenceservice-multi-node-scc": Forbidden: not usable by user or serviceaccount, provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1000}: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1001190000, 1001199999], provider "restricted-v3": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "pcap-dedicated-admi ns": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/15 17:52:15 17:49:52 INFO ReplicaSet/llmisvc-controller-manager-6dffbf4d96: SuccessfulCreate - Created pod: llmisvc-controller-manager-6dffbf4d96-x7qj8 2026/09/15 17:52:15 17:49:52 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-6dffbf4d96 from 0 to 1 2026/09/15 17:52:15 17:49:27 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7877b49576-98v5n_2020fd55-cbdb-40af-bd32-4d2568de00bf became leader 2026/09/15 17:52:15 17:49:25 INFO Lease/kserve-module-controller-manager: LeaderElection - kserve-module-controller-manager-858fdb6fd-g68rz_9a367b20-0750-4043-87b2-fa0610749845 became leader 2026/09/15 17:52:15 17:49:10 INFO Pod/kubeflow-trainer-controller-manager-7877b49576-98v5n: AddedInterface - Add eth0 [10.129.71.229/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:10 INFO Pod/kubeflow-trainer-controller-manager-7877b49576-98v5n: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:54ee8084549687b9db3e57589fb0e108df6480ecc39143d22f4900e51f5c6991" already present on machine 2026/09/15 17:52:15 17:49:10 INFO Pod/kubeflow-trainer-controller-manager-7877b49576-98v5n: Created - Created container: manager 2026/09/15 17:52:15 17:49:10 INFO Pod/kubeflow-trainer-controller-manager-7877b49576-98v5n: Started - Started container manager 2026/09/15 17:52:15 17:49:09 INFO Pod/kubeflow-trainer-controller-manager-7877b49576-98v5n: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7877b49576-98v5n to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:09 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:49:09 INFO ReplicaSet/kubeflow-trainer-controller-manager-7877b49576: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7877b49576-98v5n 2026/09/15 17:52:15 17:49:09 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7877b49576 from 0 to 1 2026/09/15 17:52:15 17:49:09 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:49:05 INFO Pod/trainer-operator-controller-manager-7c67bcdbb4-qq6tx: Created - Created container: manager 2026/09/15 17:52:15 17:49:05 INFO Pod/trainer-operator-controller-manager-7c67bcdbb4-qq6tx: Started - Started container manager 2026/09/15 17:52:15 17:49:05 INFO Pod/trainer-operator-controller-manager-7c67bcdbb4-qq6tx: Pulled - Container image "quay.io/rhoai/odh-trainer-operator-rhel9@sha256:f31b4a44e67a1607e5dd6112832567e894ea0512d9b21a53577a41f5896c77cc" already present on machine 2026/09/15 17:52:15 17:49:05 INFO Pod/kserve-module-controller-manager-858fdb6fd-g68rz: AddedInterface - Add eth0 [10.129.71.227/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:05 INFO Pod/trainer-operator-controller-manager-7c67bcdbb4-qq6tx: AddedInterface - Add eth0 [10.129.71.228/23] from ovn-kubernetes 2026/09/15 17:52:15 17:49:05 INFO Pod/kserve-module-controller-manager-858fdb6fd-g68rz: Created - Created container: manager 2026/09/15 17:52:15 17:49:05 INFO Pod/kserve-module-controller-manager-858fdb6fd-g68rz: Started - Started container manager 2026/09/15 17:52:15 17:49:05 INFO Pod/kserve-module-controller-manager-858fdb6fd-g68rz: Pulled - Container image "quay.io/rhoai/odh-kserve-module-operator-rhel9@sha256:7caa7a001cc41b2cdb15f529cde6e0a3b3bc8784e959fb935716f9d312485abc" already present on machine 2026/09/15 17:52:15 17:49:04 INFO Pod/trainer-operator-controller-manager-7c67bcdbb4-qq6tx: Scheduled - Successfully assigned redhat-ods-applications/trainer-operator-controller-manager-7c67bcdbb4-qq6tx to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:04 INFO Pod/kserve-module-controller-manager-858fdb6fd-g68rz: Scheduled - Successfully assigned redhat-ods-applications/kserve-module-controller-manager-858fdb6fd-g68rz to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:49:04 INFO ReplicaSet/kserve-module-controller-manager-858fdb6fd: SuccessfulCreate - Created pod: kserve-module-controller-manager-858fdb6fd-g68rz 2026/09/15 17:52:15 17:49:04 INFO Deployment/kserve-module-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-module-controller-manager-858fdb6fd from 0 to 1 2026/09/15 17:52:15 17:49:04 INFO Deployment/trainer-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trainer-operator-controller-manager-7c67bcdbb4 from 0 to 1 2026/09/15 17:52:15 17:49:04 INFO ReplicaSet/trainer-operator-controller-manager-7c67bcdbb4: SuccessfulCreate - Created pod: trainer-operator-controller-manager-7c67bcdbb4-qq6tx 2026/09/15 17:52:15 17:48:53 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:46 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-86644cfc8b-xr4qw_e57ae618-2b18-4515-a802-b09ab94f33ac became leader 2026/09/15 17:52:15 17:48:45 INFO Pod/kserve-module-controller-manager-858fdb6fd-thc5b: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:43 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/15 17:52:15 17:48:43 INFO Pod/kserve-localmodel-controller-manager-76664cd8c7-qfbv5: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:43 INFO Pod/kserve-controller-manager-7945ffcd-rtmm8: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:43 INFO Pod/model-serving-api-74db48b547-6j52l: Killing - Stopping container server 2026/09/15 17:52:15 17:48:43 INFO Pod/odh-model-controller-8754f79ff-v7smq: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:43 INFO Pod/llmisvc-controller-manager-6dffbf4d96-ndwfl: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:42 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-wzrxn 2026/09/15 17:52:15 17:48:42 INFO Pod/kserve-localmodelnode-agent-vqgz8: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:42 INFO Pod/kserve-localmodelnode-agent-wzrxn: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:42 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-vqgz8 2026/09/15 17:52:15 17:48:28 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw: AddedInterface - Add eth0 [10.129.71.226/23] from ovn-kubernetes 2026/09/15 17:52:15 17:48:28 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:8bd0221526c5b263c9e829b4f81af49920958b5297855c6342956fdd81e84e10" already present on machine 2026/09/15 17:52:15 17:48:28 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw: Created - Created container: manager 2026/09/15 17:52:15 17:48:28 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw: Started - Started container manager 2026/09/15 17:52:15 17:48:27 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-86644cfc8b-xr4qw to ip-10-0-1-132.us-east-2.compute.internal 2026/09/15 17:52:15 17:48:27 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-jkrbk: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:27 INFO Pod/trustyai-service-operator-controller-manager-86644cfc8b-z9g29: Killing - Stopping container manager 2026/09/15 17:52:15 17:48:27 INFO ReplicaSet/trustyai-service-operator-controller-manager-86644cfc8b: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-86644cfc8b-xr4qw 2026/09/15 17:52:15 17:48:27 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-86644cfc8b from 0 to 1 2026/09/15 17:52:15 === RESOURCE QUOTAS === 2026/09/15 17:52:15 No resource quotas found in monitored namespaces 2026/09/15 17:52:15 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/09/15 17:52:15 Evidence: event Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/kserve-localmodel-controller-manager-76664cd8c7-l2dnl: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"localmodel-webhook-server-cert\" not found"],"confidence":"medium"} 2026/09/15 17:52:15 Diagnostics complete for failed test: components controller_test.go:497: Stopping: components test failed. 2026/09/15 17:52:15 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/09/15 17:52:15 Circuit breaker: infrastructure/unknown failure recorded (2/3) controller_test.go:769: Skipping tests for services/monitoring Deletion Policy: Never. Skipping deletion tests. 2026/09/15 17:52:16 Circuit breaker: no trips during this run (threshold=3)]]>