: DataScienceCluster.datasciencecluster.opendatahub.io "default-dsc" is invalid: spec.components.modelregistry: Invalid value: "object": RegistriesNamespace is immutable when model registry is Managed { ErrStatus: { TypeMeta: {Kind: "Status", APIVersion: "v1"}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, ShardInfo: nil, }, Status: "Failure", Message: "DataScienceCluster.datasciencecluster.opendatahub.io \"default-dsc\" is invalid: spec.components.modelregistry: Invalid value: \"object\": RegistriesNamespace is immutable when model registry is Managed", Reason: "Invalid", Details: { Name: "default-dsc", Group: "datasciencecluster.opendatahub.io", Kind: "DataScienceCluster", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: \"object\": RegistriesNamespace is immutable when model registry is Managed", Field: "spec.components.modelregistry", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } occurred testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test 2026/09/14 07:18:59 TEST FAILURE DETECTED: group 2 2026/09/14 07:18:59 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/09/14 07:18:59 === CLUSTER STATE === 2026/09/14 07:18:59 Node: ip-10-0-128-184.ec2.internal 2026/09/14 07:18:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/14 07:18:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/14 07:18:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/14 07:18:59 Condition Ready: True - kubelet is posting ready status 2026/09/14 07:18:59 Resources: 7500m CPU, 31236212Ki memory allocatable (of 8000m CPU, 32387188Ki memory capacity) 2026/09/14 07:18:59 Node: ip-10-0-139-63.ec2.internal 2026/09/14 07:18:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/14 07:18:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/14 07:18:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/14 07:18:59 Condition Ready: True - kubelet is posting ready status 2026/09/14 07:18:59 Resources: 7500m CPU, 31236216Ki memory allocatable (of 8000m CPU, 32387192Ki memory capacity) 2026/09/14 07:18:59 Node: ip-10-0-140-220.ec2.internal 2026/09/14 07:18:59 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/14 07:18:59 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/14 07:18:59 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/14 07:18:59 Condition Ready: True - kubelet is posting ready status 2026/09/14 07:18:59 Resources: 7500m CPU, 31236216Ki memory allocatable (of 8000m CPU, 32387192Ki memory capacity) 2026/09/14 07:18:59 === DEPLOYMENTS === 2026/09/14 07:18:59 Namespace: redhat-ods-applications 2026/09/14 07:18:59 Deployment aihub-controller-manager: 1/1 (READY) 2026/09/14 07:18:59 Deployment catalog-controller-manager: 1/1 (READY) 2026/09/14 07:18:59 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/09/14 07:18:59 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/09/14 07:18:59 Deployment odh-observability: 1/1 (READY) 2026/09/14 07:18:59 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/09/14 07:18:59 Namespace: kube-system 2026/09/14 07:18:59 === PODS === 2026/09/14 07:18:59 Namespace redhat-ods-applications: no problematic pods found 2026/09/14 07:18:59 Namespace kube-system: no problematic pods found 2026/09/14 07:18:59 === OPERATOR STATUS === 2026/09/14 07:18:59 Operator deployment: 3/3 ready 2026/09/14 07:18:59 === DSCI STATUS === 2026/09/14 07:18:59 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; ThanosQuerierAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; NodeMetricsEndpointAvailable=False; PersesAvailable=False 2026/09/14 07:18:59 === DSC STATUS === 2026/09/14 07:18:59 DSC default-dsc: 2026/09/14 07:18:59 Ready: True - 2026/09/14 07:18:59 ProvisioningSucceeded: True - 2026/09/14 07:18:59 ComponentsReady: True - 2026/09/14 07:18:59 ModulesReady: True - 2026/09/14 07:18:59 AIGatewayReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/09/14 07:18:59 BatchGatewayReady: False - Submodule ManagementState is set to Removed 2026/09/14 07:18:59 DashboardReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 FeastOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 KserveLLMInferenceServiceDependencies: False - Submodule ManagementState is set to Removed 2026/09/14 07:18:59 KserveLLMInferenceServiceWideEPDependencies: False - Submodule ManagementState is set to Removed 2026/09/14 07:18:59 KserveReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 KueueReady: False - Component ManagementState is set to Removed 2026/09/14 07:18:59 MCPLifecycleOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 MLflowOperatorReady: True - MLflowOperator is ready to manage MLflow custom resources 2026/09/14 07:18:59 ModelRegistryReady: True - 2026/09/14 07:18:59 ModelsAsAServiceReady: False - Submodule ManagementState is set to Removed 2026/09/14 07:18:59 OGXReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 RayReady: False - Component ManagementState is set to Removed 2026/09/14 07:18:59 SparkOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 TrainerReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 TrustyAIReady: True - 2026/09/14 07:18:59 WorkbenchesReady: False - Module ManagementState is set to Removed 2026/09/14 07:18:59 WorkbenchesV2Ready: False - Submodule ManagementState is set to Removed 2026/09/14 07:18:59 === RECENT EVENTS === 2026/09/14 07:18:59 07:17:26 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:17:26 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:17:26 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:17:26 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:17:26 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:15:01 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:15:01 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:15:01 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:15:01 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:15:01 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:54 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:54 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:54 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:54 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:54 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:34 WARN Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe failed: Get "https://10.132.0.157:8943/healthcheck": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2026/09/14 07:18:59 07:14:25 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:25 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:25 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:25 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:25 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:14:25 WARN Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe failed: Get "https://10.132.0.157:8444/healthz": context deadline exceeded 2026/09/14 07:18:59 07:14:23 WARN Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/14 07:18:59 07:14:22 WARN Pod/rhods-dashboard-67b8584bc4-4cqcq: Unhealthy - Readiness probe failed: Get "https://10.134.0.97:8943/healthcheck": dial tcp 10.134.0.97:8943: connect: connection refused 2026/09/14 07:18:59 07:14:22 WARN Pod/rhods-dashboard-67b8584bc4-4cqcq: Unhealthy - Readiness probe failed: Get "https://10.134.0.97:8444/healthz": dial tcp 10.134.0.97:8444: connect: connection refused 2026/09/14 07:18:59 07:14:21 WARN Pod/rhods-dashboard-67b8584bc4-4cqcq: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/14 07:18:59 07:14:19 WARN Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe failed: Get "https://10.132.0.157:8444/healthz": dial tcp 10.132.0.157:8444: connect: connection refused 2026/09/14 07:18:59 07:14:19 WARN Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe failed: Get "https://10.132.0.157:8943/healthcheck": dial tcp 10.132.0.157:8943: connect: connection refused 2026/09/14 07:18:59 07:14:14 WARN Pod/rhods-dashboard-8765f8947-mdwmk: Unhealthy - Readiness probe failed: Get "https://10.134.0.96:8943/healthcheck": dial tcp 10.134.0.96:8943: connect: connection refused 2026/09/14 07:18:59 07:14:14 WARN Pod/rhods-dashboard-67b8584bc4-gv7ct: Unhealthy - Readiness probe failed: Get "https://10.132.0.158:8444/healthz": dial tcp 10.132.0.158:8444: connect: connection refused 2026/09/14 07:18:59 07:14:14 WARN Pod/rhods-dashboard-8765f8947-mdwmk: Unhealthy - Readiness probe failed: Get "https://10.134.0.96:8444/healthz": dial tcp 10.134.0.96:8444: connect: connection refused 2026/09/14 07:18:59 07:14:13 WARN Pod/rhods-dashboard-67b8584bc4-gv7ct: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/14 07:18:59 07:14:13 WARN Pod/rhods-dashboard-8765f8947-mdwmk: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/14 07:18:59 07:14:11 WARN Pod/rhods-dashboard-67b8584bc4-gv7ct: Unhealthy - Readiness probe failed: Get "https://10.132.0.158:8943/healthcheck": dial tcp 10.132.0.158:8943: connect: connection refused 2026/09/14 07:18:59 07:14:10 WARN Pod/rhods-dashboard-67b8584bc4-gv7ct: FailedMount - MountVolume.SetUp failed for volume "proxy-tls" : secret "dashboard-proxy-tls" not found 2026/09/14 07:18:59 07:14:10 WARN Pod/rhods-dashboard-67b8584bc4-gv7ct: FailedMount - MountVolume.SetUp failed for volume "kube-rbac-proxy-config" : configmap "kube-rbac-proxy-config" not found 2026/09/14 07:18:59 07:14:03 WARN Pod/rhods-dashboard-67b8584bc4-96c87: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/14 07:18:59 07:14:01 WARN Pod/rhods-dashboard-67b8584bc4-96c87: Unhealthy - Readiness probe failed: Get "https://10.132.0.145:8943/healthcheck": dial tcp 10.132.0.145:8943: connect: connection refused 2026/09/14 07:18:59 07:14:01 WARN Pod/rhods-dashboard-67b8584bc4-96c87: Unhealthy - Readiness probe failed: Get "https://10.132.0.145:8444/healthz": dial tcp 10.132.0.145:8444: connect: connection refused 2026/09/14 07:18:59 07:14:01 WARN Pod/rhods-dashboard-67b8584bc4-htlxr: Unhealthy - Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 2026/09/14 07:18:59 07:13:59 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:13:59 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:13:59 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:13:59 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 07:13:59 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:18:59 === RESOURCE QUOTAS === 2026/09/14 07:18:59 No resource quotas found in monitored namespaces 2026/09/14 07:18:59 Classification for group 2: infrastructure/probe-failure (code=1014, confidence=medium) 2026/09/14 07:18:59 Evidence: event Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe failed: Get "https://10.132.0.157:8943/healthcheck": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"probe-failure","error_code":1014,"evidence":["event Pod/rhods-dashboard-8765f8947-x6hn7: Unhealthy - Readiness probe failed: Get \"https://10.132.0.157:8943/healthcheck\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"],"confidence":"medium"} 2026/09/14 07:18:59 Diagnostics complete for failed test: group 2 controller_test.go:769: Stopping: group 2 test failed. 2026/09/14 07:18:59 Circuit breaker: failure classified as infrastructure/probe-failure (code=1014) 2026/09/14 07:18:59 Circuit breaker: infrastructure/unknown failure recorded (1/3) 2026-09-14T07:24:44Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.6.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.31"}} 2026/09/14 07:27:56 TEST FAILURE DETECTED: components 2026/09/14 07:27:56 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/09/14 07:27:57 === CLUSTER STATE === 2026/09/14 07:27:57 Node: ip-10-0-128-184.ec2.internal 2026/09/14 07:27:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/14 07:27:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/14 07:27:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/14 07:27:57 Condition Ready: True - kubelet is posting ready status 2026/09/14 07:27:57 Resources: 7500m CPU, 31236212Ki memory allocatable (of 8000m CPU, 32387188Ki memory capacity) 2026/09/14 07:27:57 Node: ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/14 07:27:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/14 07:27:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/14 07:27:57 Condition Ready: True - kubelet is posting ready status 2026/09/14 07:27:57 Resources: 7500m CPU, 31236216Ki memory allocatable (of 8000m CPU, 32387192Ki memory capacity) 2026/09/14 07:27:57 Node: ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/09/14 07:27:57 Condition DiskPressure: False - kubelet has no disk pressure 2026/09/14 07:27:57 Condition PIDPressure: False - kubelet has sufficient PID available 2026/09/14 07:27:57 Condition Ready: True - kubelet is posting ready status 2026/09/14 07:27:57 Resources: 7500m CPU, 31236216Ki memory allocatable (of 8000m CPU, 32387192Ki memory capacity) 2026/09/14 07:27:57 === DEPLOYMENTS === 2026/09/14 07:27:57 Namespace: redhat-ods-applications 2026/09/14 07:27:57 Deployment aihub-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment catalog-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment kserve-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment kserve-localmodel-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment kserve-module-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment model-registry-operator-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Deployment model-serving-api: 1/1 (READY) 2026/09/14 07:27:57 Deployment odh-model-controller: 1/1 (READY) 2026/09/14 07:27:57 Deployment odh-observability: 1/1 (READY) 2026/09/14 07:27:57 Deployment trainer-operator-controller-manager: 1/1 (READY) 2026/09/14 07:27:57 Namespace: kube-system 2026/09/14 07:27:57 === PODS === 2026/09/14 07:27:57 Namespace kube-system: no problematic pods found 2026/09/14 07:27:57 Namespace redhat-ods-applications: no problematic pods found 2026/09/14 07:27:57 === OPERATOR STATUS === 2026/09/14 07:27:57 Operator deployment: 3/3 ready 2026/09/14 07:27:57 === DSCI STATUS === 2026/09/14 07:27:57 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; ThanosQuerierAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; NodeMetricsEndpointAvailable=False; PersesAvailable=False 2026/09/14 07:27:57 === DSC STATUS === 2026/09/14 07:27:57 DSC default-dsc: 2026/09/14 07:27:57 Ready: True - 2026/09/14 07:27:57 ProvisioningSucceeded: True - 2026/09/14 07:27:57 ComponentsReady: True - All registered components have ManagementState Removed or are not configured 2026/09/14 07:27:57 ModulesReady: True - 2026/09/14 07:27:57 AIGatewayReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/09/14 07:27:57 BatchGatewayReady: False - Submodule ManagementState is set to Removed 2026/09/14 07:27:57 DashboardReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 FeastOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 KserveLLMInferenceServiceDependencies: True - 2026/09/14 07:27:57 KserveLLMInferenceServiceWideEPDependencies: True - 2026/09/14 07:27:57 KserveReady: True - 2026/09/14 07:27:57 KueueReady: False - Component ManagementState is set to Removed 2026/09/14 07:27:57 MCPLifecycleOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 MLflowOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 ModelRegistryReady: True - 2026/09/14 07:27:57 ModelsAsAServiceReady: False - Submodule ManagementState is set to Removed 2026/09/14 07:27:57 OGXReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 RayReady: False - Component ManagementState is set to Removed 2026/09/14 07:27:57 SparkOperatorReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 TrainerReady: True - 2026/09/14 07:27:57 TrustyAIReady: False - Component ManagementState is set to Removed 2026/09/14 07:27:57 WorkbenchesReady: False - Module ManagementState is set to Removed 2026/09/14 07:27:57 WorkbenchesV2Ready: False - Submodule ManagementState is set to Removed 2026/09/14 07:27:57 === RECENT EVENTS === 2026/09/14 07:27:57 07:27:33 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-f8979794d-x8rx4_523cfe9e-deed-49f4-b156-a528b551947a became leader 2026/09/14 07:27:57 07:27:32 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7cf7cb877d-xnsnc_8ff2d32d-933b-4712-9e9d-06dafbf2041a became leader 2026/09/14 07:27:57 07:27:32 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-75d5d85f5f-fgz89_3f86834a-65e7-4ef6-a835-bb00313bbd82 became leader 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-controller-manager-7cf7cb877d-xnsnc: Started - Started container manager 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-v2hsl: Started - Started container manager 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" in 378ms (378ms including waiting). Image size: 245939552 bytes. 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-rq72j: Created - Created container: manager 2026/09/14 07:27:57 07:27:15 INFO Pod/odh-model-controller-75d5d85f5f-fgz89: Started - Started container manager 2026/09/14 07:27:57 07:27:15 INFO Pod/model-serving-api-7b9956d474-4bm8m: Created - Created container: server 2026/09/14 07:27:57 07:27:15 INFO Pod/odh-model-controller-75d5d85f5f-fgz89: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" in 365ms (365ms including waiting). Image size: 235603578 bytes. 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-rq72j: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 313ms (313ms including waiting). Image size: 245681483 bytes. 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-rq72j: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-rq72j: AddedInterface - Add eth0 [10.134.0.118/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: AddedInterface - Add eth0 [10.134.0.116/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-v2hsl: AddedInterface - Add eth0 [10.133.0.107/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-v2hsl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-v2hsl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 320ms (320ms including waiting). Image size: 245681483 bytes. 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-v2hsl: Created - Created container: manager 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-controller-manager-7cf7cb877d-xnsnc: Created - Created container: manager 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: Created - Created container: manager 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: Started - Started container manager 2026/09/14 07:27:57 07:27:15 INFO Pod/model-serving-api-7b9956d474-4bm8m: Started - Started container server 2026/09/14 07:27:57 07:27:15 INFO Pod/odh-model-controller-75d5d85f5f-fgz89: Created - Created container: manager 2026/09/14 07:27:57 07:27:15 INFO Pod/kserve-localmodelnode-agent-rq72j: Started - Started container manager 2026/09/14 07:27:57 07:27:15 INFO Pod/model-serving-api-7b9956d474-4bm8m: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:ad1c5ec4f15df0dff60ef6d93886e674b4bc512d20da31598b84ac5a009a13b3" already present on machine 2026/09/14 07:27:57 07:27:15 INFO Pod/llmisvc-controller-manager-f8979794d-x8rx4: Created - Created container: manager 2026/09/14 07:27:57 07:27:15 INFO Pod/model-serving-api-7b9956d474-4bm8m: AddedInterface - Add eth0 [10.132.0.169/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:15 INFO Pod/llmisvc-controller-manager-f8979794d-x8rx4: Started - Started container manager 2026/09/14 07:27:57 07:27:15 INFO Pod/llmisvc-controller-manager-f8979794d-x8rx4: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" in 428ms (428ms including waiting). Image size: 235369934 bytes. 2026/09/14 07:27:57 07:27:14 INFO Pod/kserve-localmodelnode-agent-rq72j: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-rq72j to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:27:14 INFO Pod/kserve-localmodelnode-agent-v2hsl: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-v2hsl to ip-10-0-128-184.ec2.internal 2026/09/14 07:27:57 07:27:14 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:27:14 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:27:14 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:27:14 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:27:14 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:27:14 INFO Pod/odh-model-controller-75d5d85f5f-fgz89: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-75d5d85f5f-fgz89 to ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 07:27:14 INFO Pod/model-serving-api-7b9956d474-4bm8m: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-7b9956d474-4bm8m to ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 07:27:14 INFO Pod/llmisvc-controller-manager-f8979794d-x8rx4: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-f8979794d-x8rx4 to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:27:14 INFO ReplicaSet/llmisvc-controller-manager-f8979794d: SuccessfulCreate - Created pod: llmisvc-controller-manager-f8979794d-x8rx4 2026/09/14 07:27:57 07:27:14 INFO Pod/kserve-controller-manager-7cf7cb877d-xnsnc: AddedInterface - Add eth0 [10.134.0.115/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:14 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-75d5d85f5f from 0 to 1 2026/09/14 07:27:57 07:27:14 INFO ReplicaSet/odh-model-controller-75d5d85f5f: SuccessfulCreate - Created pod: odh-model-controller-75d5d85f5f-fgz89 2026/09/14 07:27:57 07:27:14 INFO Pod/odh-model-controller-75d5d85f5f-fgz89: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" 2026/09/14 07:27:57 07:27:14 INFO Pod/odh-model-controller-75d5d85f5f-fgz89: AddedInterface - Add eth0 [10.132.0.170/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:14 INFO Pod/llmisvc-controller-manager-f8979794d-x8rx4: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" 2026/09/14 07:27:57 07:27:14 INFO Pod/llmisvc-controller-manager-f8979794d-x8rx4: AddedInterface - Add eth0 [10.134.0.117/23] from ovn-kubernetes 2026/09/14 07:27:57 07:27:14 WARN Pod/model-serving-api-7b9956d474-4bm8m: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/09/14 07:27:57 07:27:14 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-7b9956d474 from 0 to 1 2026/09/14 07:27:57 07:27:14 INFO ReplicaSet/model-serving-api-7b9956d474: SuccessfulCreate - Created pod: model-serving-api-7b9956d474-4bm8m 2026/09/14 07:27:57 07:27:14 WARN Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "localmodel-webhook-server-cert" not found 2026/09/14 07:27:57 07:27:14 INFO Pod/kserve-controller-manager-7cf7cb877d-xnsnc: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" 2026/09/14 07:27:57 07:27:14 INFO Pod/kserve-controller-manager-7cf7cb877d-xnsnc: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" in 403ms (403ms including waiting). Image size: 247701913 bytes. 2026/09/14 07:27:57 07:27:14 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/14 07:27:57 07:27:14 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-v2hsl 2026/09/14 07:27:57 07:27:14 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-rq72j 2026/09/14 07:27:57 07:27:13 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodel-controller-manager-6f67bbcbf7-xd46p to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:27:13 INFO Pod/kserve-controller-manager-7cf7cb877d-xnsnc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7cf7cb877d-xnsnc to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:27:13 INFO Deployment/kserve-localmodel-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-localmodel-controller-manager-6f67bbcbf7 from 0 to 1 2026/09/14 07:27:57 07:27:13 INFO ReplicaSet/kserve-controller-manager-7cf7cb877d: SuccessfulCreate - Created pod: kserve-controller-manager-7cf7cb877d-xnsnc 2026/09/14 07:27:57 07:27:13 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7cf7cb877d from 0 to 1 2026/09/14 07:27:57 07:27:13 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-f8979794d from 0 to 1 2026/09/14 07:27:57 07:27:13 INFO ReplicaSet/kserve-localmodel-controller-manager-6f67bbcbf7: SuccessfulCreate - Created pod: kserve-localmodel-controller-manager-6f67bbcbf7-xd46p 2026/09/14 07:27:57 07:26:49 INFO Lease/kserve-module-controller-manager: LeaderElection - kserve-module-controller-manager-658f689cf8-vxlvh_d2b5fc47-8de7-41eb-a5d4-01679be0f75f became leader 2026/09/14 07:27:57 07:26:28 INFO Pod/kserve-module-controller-manager-658f689cf8-vxlvh: Pulled - Container image "quay.io/rhoai/odh-kserve-module-operator-rhel9@sha256:7caa7a001cc41b2cdb15f529cde6e0a3b3bc8784e959fb935716f9d312485abc" already present on machine 2026/09/14 07:27:57 07:26:28 INFO Pod/kserve-module-controller-manager-658f689cf8-vxlvh: Created - Created container: manager 2026/09/14 07:27:57 07:26:28 INFO Pod/kserve-module-controller-manager-658f689cf8-vxlvh: Started - Started container manager 2026/09/14 07:27:57 07:26:28 INFO Pod/kserve-module-controller-manager-658f689cf8-vxlvh: AddedInterface - Add eth0 [10.134.0.114/23] from ovn-kubernetes 2026/09/14 07:27:57 07:26:27 INFO Pod/kserve-module-controller-manager-658f689cf8-vxlvh: Scheduled - Successfully assigned redhat-ods-applications/kserve-module-controller-manager-658f689cf8-vxlvh to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:26:27 INFO Deployment/kserve-module-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-module-controller-manager-658f689cf8 from 0 to 1 2026/09/14 07:27:57 07:26:27 INFO ReplicaSet/kserve-module-controller-manager-658f689cf8: SuccessfulCreate - Created pod: kserve-module-controller-manager-658f689cf8-vxlvh 2026/09/14 07:27:57 07:26:22 INFO Pod/kserve-module-controller-manager-658f689cf8-xs9s4: Killing - Stopping container manager 2026/09/14 07:27:57 07:26:21 INFO Pod/model-serving-api-7b9956d474-2lnqq: Killing - Stopping container server 2026/09/14 07:27:57 07:26:21 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: Killing - Stopping container manager 2026/09/14 07:27:57 07:26:21 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: Killing - Stopping container manager 2026/09/14 07:27:57 07:26:21 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: Killing - Stopping container manager 2026/09/14 07:27:57 07:26:21 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: Killing - Stopping container manager 2026/09/14 07:27:57 07:26:20 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:26:20 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:26:20 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:26:20 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:26:20 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:26:20 INFO Pod/kserve-localmodelnode-agent-mth4c: Killing - Stopping container manager 2026/09/14 07:27:57 07:26:20 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-mth4c 2026/09/14 07:27:57 07:26:20 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-frjv9 2026/09/14 07:27:57 07:26:20 INFO Pod/kserve-localmodelnode-agent-frjv9: Killing - Stopping container manager 2026/09/14 07:27:57 07:25:56 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7cf7cb877d-c55gk_5b4a9433-95dc-455b-8017-6cdf55b97e8e became leader 2026/09/14 07:27:57 07:25:55 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-f8979794d-xjc9r_d7eae0e9-c2d0-462c-a261-ec15a095d6cf became leader 2026/09/14 07:27:57 07:25:52 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-75d5d85f5f-kjqlt_fa9cfdae-40db-4d72-98fd-04c70c0cd350 became leader 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-frjv9: AddedInterface - Add eth0 [10.134.0.113/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-frjv9: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 513ms (513ms including waiting). Image size: 245681483 bytes. 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-frjv9: Created - Created container: manager 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-mth4c: Started - Started container manager 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-mth4c: Created - Created container: manager 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-mth4c: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" in 419ms (419ms including waiting). Image size: 245681483 bytes. 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-mth4c: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-frjv9: Started - Started container manager 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-mth4c: AddedInterface - Add eth0 [10.133.0.106/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:50 INFO Pod/kserve-localmodelnode-agent-frjv9: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodelnode-agent-rhel9@sha256:45fb7665fc66de0514d764d13759c9c11e54758e1fed020a23617085a963052b" 2026/09/14 07:27:57 07:25:49 INFO Pod/kserve-localmodelnode-agent-frjv9: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-frjv9 to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:25:49 INFO Pod/kserve-localmodelnode-agent-mth4c: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodelnode-agent-mth4c to ip-10-0-128-184.ec2.internal 2026/09/14 07:27:57 07:25:37 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: Started - Started container manager 2026/09/14 07:27:57 07:25:37 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: Created - Created container: manager 2026/09/14 07:27:57 07:25:37 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: Created - Created container: manager 2026/09/14 07:27:57 07:25:37 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" in 508ms (508ms including waiting). Image size: 235603578 bytes. 2026/09/14 07:27:57 07:25:37 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: Started - Started container manager 2026/09/14 07:27:57 07:25:37 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: Started - Started container manager 2026/09/14 07:27:57 07:25:37 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: Created - Created container: manager 2026/09/14 07:27:57 07:25:36 WARN Pod/kserve-localmodelnode-agent-frjv9: FailedScheduling - 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 2026/09/14 07:27:57 07:25:36 WARN Pod/kserve-localmodelnode-agent-mth4c: FailedScheduling - 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found 2026/09/14 07:27:57 07:25:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:25:36 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" 2026/09/14 07:27:57 07:25:36 INFO Pod/model-serving-api-7b9956d474-2lnqq: AddedInterface - Add eth0 [10.132.0.168/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:36 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: AddedInterface - Add eth0 [10.134.0.112/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: Started - Started container manager 2026/09/14 07:27:57 07:25:36 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-mth4c 2026/09/14 07:27:57 07:25:36 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:a92306bca53a39972e87df505dd66c7165a1e2bbd39b0ca49881db790c883006" 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: AddedInterface - Add eth0 [10.134.0.110/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:36 INFO Pod/model-serving-api-7b9956d474-2lnqq: Started - Started container server 2026/09/14 07:27:57 07:25:36 INFO Pod/model-serving-api-7b9956d474-2lnqq: Created - Created container: server 2026/09/14 07:27:57 07:25:36 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "openshift-ai-inferenceservice-image-volume-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "openshift-ai-llminferenceservice-scc": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "kserve-localmodel-permissions-scc": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/14 07:27:57 07:25:36 INFO Pod/model-serving-api-7b9956d474-2lnqq: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:ad1c5ec4f15df0dff60ef6d93886e674b4bc512d20da31598b84ac5a009a13b3" already present on machine 2026/09/14 07:27:57 07:25:36 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: AddedInterface - Add eth0 [10.134.0.111/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:36 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulCreate - Created pod: kserve-localmodelnode-agent-frjv9 2026/09/14 07:27:57 07:25:36 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:9e7ffbc0d6b474aa759feb6409de58d841a32e25e5e2e86d7b923bc754d84272" in 462ms (462ms including waiting). Image size: 235369934 bytes. 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: Pulling - Pulling image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: Created - Created container: manager 2026/09/14 07:27:57 07:25:36 WARN DaemonSet/kserve-localmodelnode-agent: FailedCreate - Error creating: pods "kserve-localmodelnode-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: [1000]: 1000 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1000: must be in the ranges: [1000620000, 1000629999], provider restricted-v3: .spec.securityContext.hostUsers: Invalid value: null: Host Users must be set to false, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nested-container": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "insights-runtime-extractor-scc": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" in 479ms (479ms including waiting). Image size: 247701913 bytes. 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-localmodel-controller-rhel9@sha256:c62459337c34a180b176be09241ca2e68b1ac9884b2e2fe5e953a3bb384e3e10" in 573ms (573ms including waiting). Image size: 245939552 bytes. 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:c90681bafc90b753b72f71ce873899fd304a44af65f92605e3283de7bae7282c" 2026/09/14 07:27:57 07:25:36 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: AddedInterface - Add eth0 [10.134.0.109/23] from ovn-kubernetes 2026/09/14 07:27:57 07:25:35 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:25:35 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:25:35 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:25:35 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:25:35 INFO Pod/odh-model-controller-75d5d85f5f-kjqlt: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-75d5d85f5f-kjqlt to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:25:35 INFO Pod/model-serving-api-7b9956d474-2lnqq: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-7b9956d474-2lnqq to ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 07:25:35 INFO Pod/llmisvc-controller-manager-f8979794d-xjc9r: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-f8979794d-xjc9r to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:25:35 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg: Scheduled - Successfully assigned redhat-ods-applications/kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:25:35 INFO Pod/kserve-controller-manager-7cf7cb877d-c55gk: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7cf7cb877d-c55gk to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:25:35 INFO ReplicaSet/llmisvc-controller-manager-f8979794d: SuccessfulCreate - Created pod: llmisvc-controller-manager-f8979794d-xjc9r 2026/09/14 07:27:57 07:25:35 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7cf7cb877d from 0 to 1 2026/09/14 07:27:57 07:25:35 INFO ReplicaSet/model-serving-api-7b9956d474: SuccessfulCreate - Created pod: model-serving-api-7b9956d474-2lnqq 2026/09/14 07:27:57 07:25:35 INFO ReplicaSet/odh-model-controller-75d5d85f5f: SuccessfulCreate - Created pod: odh-model-controller-75d5d85f5f-kjqlt 2026/09/14 07:27:57 07:25:35 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-7b9956d474 from 0 to 1 2026/09/14 07:27:57 07:25:35 INFO ReplicaSet/kserve-controller-manager-7cf7cb877d: SuccessfulCreate - Created pod: kserve-controller-manager-7cf7cb877d-c55gk 2026/09/14 07:27:57 07:25:35 INFO ReplicaSet/kserve-localmodel-controller-manager-6f67bbcbf7: SuccessfulCreate - Created pod: kserve-localmodel-controller-manager-6f67bbcbf7-wf6bg 2026/09/14 07:27:57 07:25:35 INFO Deployment/kserve-localmodel-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-localmodel-controller-manager-6f67bbcbf7 from 0 to 1 2026/09/14 07:27:57 07:25:35 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-75d5d85f5f from 0 to 1 2026/09/14 07:27:57 07:25:35 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-f8979794d from 0 to 1 2026/09/14 07:27:57 07:25:13 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-8f984c59c-c2tl7_f8bf75be-77fa-4123-a606-b6b2fe60d9a6 became leader 2026/09/14 07:27:57 07:25:10 INFO Lease/kserve-module-controller-manager: LeaderElection - kserve-module-controller-manager-658f689cf8-xs9s4_dba700c2-b415-4b15-8c14-6b9c347ce95f became leader 2026/09/14 07:27:57 07:24:55 INFO Pod/kubeflow-trainer-controller-manager-8f984c59c-c2tl7: Started - Started container manager 2026/09/14 07:27:57 07:24:55 INFO Pod/kubeflow-trainer-controller-manager-8f984c59c-c2tl7: Created - Created container: manager 2026/09/14 07:27:57 07:24:55 INFO Pod/kubeflow-trainer-controller-manager-8f984c59c-c2tl7: Pulled - Successfully pulled image "quay.io/rhoai/odh-trainer-rhel9@sha256:54ee8084549687b9db3e57589fb0e108df6480ecc39143d22f4900e51f5c6991" in 1.565s (1.565s including waiting). Image size: 158408920 bytes. 2026/09/14 07:27:57 07:24:54 INFO Pod/kubeflow-trainer-controller-manager-8f984c59c-c2tl7: AddedInterface - Add eth0 [10.132.0.167/23] from ovn-kubernetes 2026/09/14 07:27:57 07:24:54 INFO Pod/kubeflow-trainer-controller-manager-8f984c59c-c2tl7: Pulling - Pulling image "quay.io/rhoai/odh-trainer-rhel9@sha256:54ee8084549687b9db3e57589fb0e108df6480ecc39143d22f4900e51f5c6991" 2026/09/14 07:27:57 07:24:53 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:53 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:53 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:53 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:53 INFO Pod/kubeflow-trainer-controller-manager-8f984c59c-c2tl7: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-8f984c59c-c2tl7 to ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 07:24:53 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-8f984c59c from 0 to 1 2026/09/14 07:27:57 07:24:53 INFO ReplicaSet/kubeflow-trainer-controller-manager-8f984c59c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-8f984c59c-c2tl7 2026/09/14 07:27:57 07:24:49 INFO Pod/trainer-operator-controller-manager-d6bd7f-j2nqf: Started - Started container manager 2026/09/14 07:27:57 07:24:49 INFO Pod/trainer-operator-controller-manager-d6bd7f-j2nqf: Created - Created container: manager 2026/09/14 07:27:57 07:24:49 INFO Pod/trainer-operator-controller-manager-d6bd7f-j2nqf: Pulled - Successfully pulled image "quay.io/rhoai/odh-trainer-operator-rhel9@sha256:f31b4a44e67a1607e5dd6112832567e894ea0512d9b21a53577a41f5896c77cc" in 1.991s (1.991s including waiting). Image size: 219806744 bytes. 2026/09/14 07:27:57 07:24:47 INFO Pod/trainer-operator-controller-manager-d6bd7f-j2nqf: AddedInterface - Add eth0 [10.132.0.166/23] from ovn-kubernetes 2026/09/14 07:27:57 07:24:47 INFO Pod/kserve-module-controller-manager-658f689cf8-xs9s4: Created - Created container: manager 2026/09/14 07:27:57 07:24:47 INFO Pod/trainer-operator-controller-manager-d6bd7f-j2nqf: Pulling - Pulling image "quay.io/rhoai/odh-trainer-operator-rhel9@sha256:f31b4a44e67a1607e5dd6112832567e894ea0512d9b21a53577a41f5896c77cc" 2026/09/14 07:27:57 07:24:47 INFO Pod/kserve-module-controller-manager-658f689cf8-xs9s4: AddedInterface - Add eth0 [10.134.0.108/23] from ovn-kubernetes 2026/09/14 07:27:57 07:24:47 INFO Pod/kserve-module-controller-manager-658f689cf8-xs9s4: Pulled - Container image "quay.io/rhoai/odh-kserve-module-operator-rhel9@sha256:7caa7a001cc41b2cdb15f529cde6e0a3b3bc8784e959fb935716f9d312485abc" already present on machine 2026/09/14 07:27:57 07:24:47 INFO Pod/kserve-module-controller-manager-658f689cf8-xs9s4: Started - Started container manager 2026/09/14 07:27:57 07:24:46 INFO Pod/trainer-operator-controller-manager-d6bd7f-j2nqf: Scheduled - Successfully assigned redhat-ods-applications/trainer-operator-controller-manager-d6bd7f-j2nqf to ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 07:24:46 INFO Pod/kserve-module-controller-manager-658f689cf8-xs9s4: Scheduled - Successfully assigned redhat-ods-applications/kserve-module-controller-manager-658f689cf8-xs9s4 to ip-10-0-139-63.ec2.internal 2026/09/14 07:27:57 07:24:46 INFO ReplicaSet/kserve-module-controller-manager-658f689cf8: SuccessfulCreate - Created pod: kserve-module-controller-manager-658f689cf8-xs9s4 2026/09/14 07:27:57 07:24:46 INFO ReplicaSet/trainer-operator-controller-manager-d6bd7f: SuccessfulCreate - Created pod: trainer-operator-controller-manager-d6bd7f-j2nqf 2026/09/14 07:27:57 07:24:46 INFO Deployment/trainer-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trainer-operator-controller-manager-d6bd7f from 0 to 1 2026/09/14 07:27:57 07:24:46 INFO Deployment/kserve-module-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-module-controller-manager-658f689cf8 from 0 to 1 2026/09/14 07:27:57 07:24:34 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:27 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-55944c6dc5-rsmgq_7cdde5c5-c7f7-47f8-9790-4bbec19c8ea9 became leader 2026/09/14 07:27:57 07:24:27 INFO Pod/kserve-module-controller-manager-658f689cf8-t26jz: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:25 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:25 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:25 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:25 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:25 INFO Pod/llmisvc-controller-manager-f8979794d-9r5r5: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:25 INFO Pod/model-serving-api-7b9956d474-dfkpf: Killing - Stopping container server 2026/09/14 07:27:57 07:24:25 INFO Pod/odh-model-controller-75d5d85f5f-p5tcm: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:25 INFO Pod/kserve-localmodel-controller-manager-6f67bbcbf7-9cfql: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:25 INFO Pod/kserve-controller-manager-7cf7cb877d-67plq: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:24 WARN ServiceMonitor/trustyai-service-operator-service-monitor: InvalidConfiguration - "redhat-ods-applications/trustyai-service-operator-service-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:24 WARN ServiceMonitor/maas-controller-metrics: InvalidConfiguration - "redhat-ods-applications/maas-controller-metrics" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:24 WARN ServiceMonitor/aihub-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/aihub-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:24 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:24 WARN ServiceMonitor/model-registry-operator-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/model-registry-operator-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/09/14 07:27:57 07:24:24 INFO Pod/kserve-localmodelnode-agent-ft4pz: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:24 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-ft4pz 2026/09/14 07:27:57 07:24:24 INFO DaemonSet/kserve-localmodelnode-agent: SuccessfulDelete - Deleted pod: kserve-localmodelnode-agent-jwtt4 2026/09/14 07:27:57 07:24:24 INFO Pod/kserve-localmodelnode-agent-jwtt4: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:09 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq to ip-10-0-140-220.ec2.internal 2026/09/14 07:27:57 07:24:09 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq: Created - Created container: manager 2026/09/14 07:27:57 07:24:09 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq: Started - Started container manager 2026/09/14 07:27:57 07:24:09 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:8bd0221526c5b263c9e829b4f81af49920958b5297855c6342956fdd81e84e10" already present on machine 2026/09/14 07:27:57 07:24:09 INFO ReplicaSet/trustyai-service-operator-controller-manager-55944c6dc5: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-55944c6dc5-rsmgq 2026/09/14 07:27:57 07:24:09 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-55944c6dc5 from 0 to 1 2026/09/14 07:27:57 07:24:09 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-rsmgq: AddedInterface - Add eth0 [10.132.0.165/23] from ovn-kubernetes 2026/09/14 07:27:57 07:24:08 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-s44ch: Killing - Stopping container manager 2026/09/14 07:27:57 07:24:08 INFO Pod/trustyai-service-operator-controller-manager-55944c6dc5-k5gkx: Killing - Stopping container manager 2026/09/14 07:27:57 === RESOURCE QUOTAS === 2026/09/14 07:27:57 No resource quotas found in monitored namespaces 2026/09/14 07:27:57 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/09/14 07:27:57 Evidence: event Pod/model-serving-api-7b9956d474-4bm8m: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/model-serving-api-7b9956d474-4bm8m: FailedMount - MountVolume.SetUp failed for volume \"tls-certs\" : secret \"model-serving-api-tls\" not found"],"confidence":"medium"} 2026/09/14 07:27:57 Diagnostics complete for failed test: components controller_test.go:497: Stopping: components test failed. 2026/09/14 07:27:57 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/09/14 07:27:57 Circuit breaker: infrastructure/unknown failure recorded (2/3) controller_test.go:769: Skipping tests for services/monitoring Deletion Policy: Never. Skipping deletion tests. 2026/09/14 07:27:58 Circuit breaker: no trips during this run (threshold=3)]]>