=== RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.03s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve test_context_test.go:223: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:233 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.1] creationTimestamp:2026-07-15T11:47:19Z generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:spec:map[f:modelsAsService:map[f:managementState:map[]] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[f:managementState:map[]]]] manager:datasciencecluster operation:Apply time:2026-07-15T11:52:33Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"9013f03c-d6c8-43c1-bdd4-c3e666f6094f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-15T11:58:15Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:9013f03c-d6c8-43c1-bdd4-c3e666f6094f]] resourceVersion:33652591 uid:e2259c28-98f9-4afa-af3b-ad74f3b1037c] spec:map[modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-15T11:58:14Z status:True type:Ready] map[lastTransitionTime:2026-07-15T11:58:15Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-15T11:58:15Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-15T11:58:14Z status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-15T11:58:14Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-15T11:58:14Z status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (683.33s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d9bnbovs84ic7b3dmq8g) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1811: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (30.87s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {([ "BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob" ] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:17fb2088-7078-41a3-bb49-94b69b392448 platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.1] creationTimestamp:2026-07-15T08:32:22Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-15T11:58:37Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-15T08:33:03Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-15T08:32:22Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-15T11:58:37Z]] name:cluster resourceVersion:33653182 uid:77466d09-c2c5-45bb-b961-9f81a442e3a1] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-15T08:33:03Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-15T08:33:03Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-15T08:33:03Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-15T08:32:44Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: ([ "BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob" ] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (31.07s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/15 11:58:26 TEST FAILURE DETECTED: group 1 2026/07/15 11:58:26 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/15 11:58:26 === CLUSTER STATE === 2026/07/15 11:58:26 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 11:58:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 11:58:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 11:58:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 11:58:26 Condition Ready: True - kubelet is posting ready status 2026/07/15 11:58:26 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 11:58:26 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 11:58:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 11:58:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 11:58:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 11:58:26 Condition Ready: True - kubelet is posting ready status 2026/07/15 11:58:26 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 11:58:26 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 11:58:26 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 11:58:26 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 11:58:26 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 11:58:26 Condition Ready: True - kubelet is posting ready status 2026/07/15 11:58:26 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 11:58:26 === DEPLOYMENTS === 2026/07/15 11:58:26 Namespace: redhat-ods-applications 2026/07/15 11:58:26 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 11:58:26 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 11:58:26 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 11:58:26 Deployment maas-api: 1/1 (READY) 2026/07/15 11:58:26 Deployment maas-controller: 1/1 (READY) 2026/07/15 11:58:26 Deployment maas-postgres: 1/1 (READY) 2026/07/15 11:58:26 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/15 11:58:26 Deployment model-serving-api: 1/1 (READY) 2026/07/15 11:58:26 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 11:58:26 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/15 11:58:26 Namespace: kube-system 2026/07/15 11:58:26 === PODS === 2026/07/15 11:58:26 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-2cllm: container manager restarts=2 2026/07/15 11:58:26 === OPERATOR STATUS === 2026/07/15 11:58:26 Operator deployment: 3/3 ready 2026/07/15 11:58:26 === DSCI STATUS === 2026/07/15 11:58:26 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 11:58:26 === DSC STATUS === 2026/07/15 11:58:26 DSC default-dsc: 2026/07/15 11:58:26 Ready: True - 2026/07/15 11:58:26 ProvisioningSucceeded: True - 2026/07/15 11:58:26 ComponentsReady: True - 2026/07/15 11:58:26 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 KserveLLMInferenceServiceDependencies: True - 2026/07/15 11:58:26 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 11:58:26 KserveReady: True - 2026/07/15 11:58:26 KueueReady: True - 2026/07/15 11:58:26 MLflowOperatorReady: True - 2026/07/15 11:58:26 ModelControllerReady: True - 2026/07/15 11:58:26 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/15 11:58:26 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 RayReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 TrainerReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 TrustyAIReady: True - 2026/07/15 11:58:26 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 11:58:26 === RECENT EVENTS === 2026/07/15 11:58:26 No recent events found in monitored namespaces 2026/07/15 11:58:26 === RESOURCE QUOTAS === 2026/07/15 11:58:26 No resource quotas found in monitored namespaces 2026/07/15 11:58:26 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/15 11:58:26 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/15 11:58:26 Diagnostics complete for failed test: group 1 controller_test.go:752: Stopping: group 1 test failed. 2026/07/15 11:58:26 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/15 11:58:26 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/15 12:01:49 TEST FAILURE DETECTED: group 2 2026/07/15 12:01:49 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/15 12:01:49 === CLUSTER STATE === 2026/07/15 12:01:49 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 12:01:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:01:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:01:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:01:49 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:01:49 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 12:01:49 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:01:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:01:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:01:49 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:01:49 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 12:01:49 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 12:01:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:01:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:01:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:01:49 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:01:49 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 12:01:49 === DEPLOYMENTS === 2026/07/15 12:01:49 Namespace: redhat-ods-applications 2026/07/15 12:01:49 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 12:01:49 Deployment maas-postgres: 1/1 (READY) 2026/07/15 12:01:49 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/15 12:01:49 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/15 12:01:49 Namespace: kube-system 2026/07/15 12:01:49 === PODS === 2026/07/15 12:01:49 Failed to collect pod data: redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-2cllm: container manager restarts=2 2026/07/15 12:01:49 === OPERATOR STATUS === 2026/07/15 12:01:49 Operator deployment: 3/3 ready 2026/07/15 12:01:49 === DSCI STATUS === 2026/07/15 12:01:49 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 12:01:49 === DSC STATUS === 2026/07/15 12:01:49 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/15 12:01:49 === RECENT EVENTS === 2026/07/15 12:01:49 12:01:48 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:47 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:42 INFO Pod/model-registry-operator-controller-manager-5797446674-csvbn: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:42 INFO Pod/model-serving-api-67c44bccdf-dhp25: Killing - Stopping container server 2026/07/15 12:01:49 12:01:42 INFO Pod/odh-model-controller-bc67559db-sdlpw: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:42 INFO Pod/maas-api-5db58d77cb-2t87h: Killing - Stopping container maas-api 2026/07/15 12:01:49 12:01:42 INFO Pod/maas-controller-85c784d7b6-cxmzr: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:41 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:41 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-hr8jx: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:41 INFO Pod/kserve-controller-manager-7747874bf-xm7sx: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:41 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:40 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:40 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:40 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:40 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 12:01:26 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bc67559db-sdlpw_f679d97d-e60e-48cc-b988-ff85688a76ab became leader 2026/07/15 12:01:49 12:01:25 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m_421c6f51-654d-4702-979c-01d6c1138b43 became leader 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 12:01:24 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7cf6dfb74c from 0 to 1 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m: AddedInterface - Add eth0 [10.129.1.154/23] from ovn-kubernetes 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m: Started - Started container manager 2026/07/15 12:01:49 12:01:24 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7cf6dfb74c: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m: Created - Created container: manager 2026/07/15 12:01:49 12:01:24 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-cxw9m: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:2fbda5dad116cec55302ab775c209ddfc87289adb653f2fe74bc8f5c02799170" already present on machine 2026/07/15 12:01:49 12:01:10 INFO Pod/odh-model-controller-bc67559db-sdlpw: Started - Started container manager 2026/07/15 12:01:49 12:01:10 INFO Pod/odh-model-controller-bc67559db-sdlpw: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 510ms (510ms including waiting). Image size: 238719886 bytes. 2026/07/15 12:01:49 12:01:10 INFO Pod/odh-model-controller-bc67559db-sdlpw: Created - Created container: manager 2026/07/15 12:01:49 12:01:09 INFO Pod/odh-model-controller-bc67559db-sdlpw: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bc67559db-sdlpw to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 12:01:09 INFO Pod/odh-model-controller-bc67559db-hp98s: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:09 INFO ReplicaSet/odh-model-controller-bc67559db: SuccessfulCreate - Created pod: odh-model-controller-bc67559db-sdlpw 2026/07/15 12:01:49 12:01:09 INFO Pod/odh-model-controller-bc67559db-sdlpw: AddedInterface - Add eth0 [10.129.1.153/23] from ovn-kubernetes 2026/07/15 12:01:49 12:01:09 INFO Pod/odh-model-controller-bc67559db-sdlpw: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 12:01:49 12:01:08 INFO Pod/odh-model-controller-bc67559db-ch8jk: Killing - Stopping container manager 2026/07/15 12:01:49 12:01:08 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bc67559db from 0 to 1 2026/07/15 12:01:49 12:00:54 INFO Pod/model-serving-api-67c44bccdf-dhp25: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 12:01:49 12:00:54 INFO Pod/model-serving-api-67c44bccdf-dhp25: Started - Started container server 2026/07/15 12:01:49 12:00:54 INFO Pod/model-serving-api-67c44bccdf-dhp25: Created - Created container: server 2026/07/15 12:01:49 12:00:54 INFO Pod/model-serving-api-67c44bccdf-dhp25: AddedInterface - Add eth0 [10.129.1.152/23] from ovn-kubernetes 2026/07/15 12:01:49 12:00:53 INFO Pod/model-serving-api-67c44bccdf-dhp25: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-dhp25 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 12:00:53 INFO Pod/model-serving-api-67c44bccdf-vtwf8: Killing - Stopping container server 2026/07/15 12:01:49 12:00:53 INFO Pod/model-serving-api-67c44bccdf-dfq72: Killing - Stopping container server 2026/07/15 12:01:49 12:00:53 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-dhp25 2026/07/15 12:01:49 12:00:53 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 12:01:49 12:00:19 INFO Job/maas-api-key-cleanup-29735280: Completed - Job completed 2026/07/15 12:01:49 12:00:19 INFO CronJob/maas-api-key-cleanup: SawCompletedJob - Saw completed job: maas-api-key-cleanup-29735280, condition: Complete 2026/07/15 12:01:49 12:00:01 INFO Pod/maas-api-key-cleanup-29735280-8qx7n: Started - Started container cleanup 2026/07/15 12:01:49 12:00:01 INFO Pod/maas-api-key-cleanup-29735280-8qx7n: Created - Created container: cleanup 2026/07/15 12:01:49 12:00:00 INFO Pod/maas-api-key-cleanup-29735280-8qx7n: Scheduled - Successfully assigned redhat-ods-applications/maas-api-key-cleanup-29735280-8qx7n to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 12:00:00 INFO Job/maas-api-key-cleanup-29735280: SuccessfulCreate - Created pod: maas-api-key-cleanup-29735280-8qx7n 2026/07/15 12:01:49 12:00:00 INFO Pod/maas-api-key-cleanup-29735280-8qx7n: Pulled - Container image "registry.redhat.io/ubi9/ubi-minimal@sha256:907b68736aa798b2d38255b7aa070b2a70acb90803864a40f05d0ec47556ddd0" already present on machine 2026/07/15 12:01:49 12:00:00 INFO Pod/maas-api-key-cleanup-29735280-8qx7n: AddedInterface - Add eth0 [10.129.1.150/23] from ovn-kubernetes 2026/07/15 12:01:49 12:00:00 INFO CronJob/maas-api-key-cleanup: SuccessfulCreate - Created job maas-api-key-cleanup-29735280 2026/07/15 12:01:49 11:59:53 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 12:01:49 11:59:53 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:2fbda5dad116cec55302ab775c209ddfc87289adb653f2fe74bc8f5c02799170" already present on machine 2026/07/15 12:01:49 11:59:53 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w: AddedInterface - Add eth0 [10.130.1.147/23] from ovn-kubernetes 2026/07/15 12:01:49 11:59:53 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w: Created - Created container: manager 2026/07/15 12:01:49 11:59:53 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w: Started - Started container manager 2026/07/15 12:01:49 11:59:53 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7cf6dfb74c: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7cf6dfb74c-s8k7w 2026/07/15 12:01:49 11:59:52 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7cf6dfb74c from 1 to 2 2026/07/15 12:01:49 11:59:40 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm_74d9e260-e119-48b2-b2c5-5baf875243fc became leader 2026/07/15 12:01:49 11:59:34 INFO Pod/odh-model-controller-bc67559db-hp98s: Created - Created container: manager 2026/07/15 12:01:49 11:59:34 INFO Pod/odh-model-controller-bc67559db-hp98s: Started - Started container manager 2026/07/15 12:01:49 11:59:33 INFO Pod/odh-model-controller-bc67559db-hp98s: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 460ms (460ms including waiting). Image size: 238719886 bytes. 2026/07/15 12:01:49 11:59:33 INFO Pod/odh-model-controller-bc67559db-hp98s: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 12:01:49 11:59:33 INFO Pod/odh-model-controller-bc67559db-hp98s: AddedInterface - Add eth0 [10.130.1.145/23] from ovn-kubernetes 2026/07/15 12:01:49 11:59:32 INFO Pod/odh-model-controller-bc67559db-hp98s: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bc67559db-hp98s to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 12:01:49 11:59:32 INFO ReplicaSet/odh-model-controller-bc67559db: SuccessfulCreate - Created pod: odh-model-controller-bc67559db-hp98s 2026/07/15 12:01:49 11:59:32 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bc67559db from 1 to 2 2026/07/15 12:01:49 11:59:13 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 11:59:13 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 11:59:13 INFO Pod/model-serving-api-67c44bccdf-dfq72: Started - Started container server 2026/07/15 12:01:49 11:59:13 INFO Pod/model-serving-api-67c44bccdf-dfq72: AddedInterface - Add eth0 [10.130.1.142/23] from ovn-kubernetes 2026/07/15 12:01:49 11:59:13 INFO Pod/model-serving-api-67c44bccdf-dfq72: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 12:01:49 11:59:13 INFO Pod/model-serving-api-67c44bccdf-dfq72: Created - Created container: server 2026/07/15 12:01:49 11:59:12 INFO Pod/model-serving-api-67c44bccdf-dfq72: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-dfq72 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 12:01:49 11:59:12 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-dfq72 2026/07/15 12:01:49 11:59:12 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 1 to 2 2026/07/15 12:01:49 11:58:58 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bc67559db-ch8jk_9584a5d1-36fa-43bd-ac33-e35860540781 became leader 2026/07/15 12:01:49 11:58:50 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 11:58:50 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 11:58:47 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-5797446674-csvbn_b6f68444-7eac-45e2-9e47-c657831ccbb9 became leader 2026/07/15 12:01:49 11:58:41 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulDelete - Deleted pod: odh-model-controller-5b978c765f-mtd9l 2026/07/15 12:01:49 11:58:41 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled down replica set odh-model-controller-5b978c765f from 1 to 0 2026/07/15 12:01:49 11:58:41 INFO Pod/odh-model-controller-5b978c765f-mtd9l: Killing - Stopping container manager 2026/07/15 12:01:49 11:58:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 11:58:32 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: authorization: failed to get authorization token [REDACTED] type "": unable to get secret "workload-variant-autoscaler-metrics-reader-token": secrets "workload-variant-autoscaler-metrics-reader-token" not found 2026/07/15 12:01:49 11:58:32 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:01:49 11:58:32 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 11:58:32 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm: AddedInterface - Add eth0 [10.129.1.147/23] from ovn-kubernetes 2026/07/15 12:01:49 11:58:32 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:2fbda5dad116cec55302ab775c209ddfc87289adb653f2fe74bc8f5c02799170" already present on machine 2026/07/15 12:01:49 11:58:32 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm: Created - Created container: manager 2026/07/15 12:01:49 11:58:32 INFO Pod/workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm: Started - Started container manager 2026/07/15 12:01:49 11:58:32 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7cf6dfb74c: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7cf6dfb74c-bpmgm 2026/07/15 12:01:49 11:58:32 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7cf6dfb74c from 0 to 1 2026/07/15 12:01:49 11:58:30 INFO Pod/model-registry-operator-controller-manager-5797446674-csvbn: Created - Created container: manager 2026/07/15 12:01:49 11:58:30 INFO Pod/model-registry-operator-controller-manager-5797446674-csvbn: Started - Started container manager 2026/07/15 12:01:49 11:58:30 INFO Pod/odh-model-controller-bc67559db-ch8jk: Started - Started container manager 2026/07/15 12:01:49 11:58:30 INFO Pod/odh-model-controller-bc67559db-ch8jk: Created - Created container: manager 2026/07/15 12:01:49 11:58:30 INFO Pod/odh-model-controller-bc67559db-ch8jk: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 510ms (510ms including waiting). Image size: 238719886 bytes. 2026/07/15 12:01:49 11:58:30 INFO Pod/model-registry-operator-controller-manager-5797446674-csvbn: AddedInterface - Add eth0 [10.129.1.146/23] from ovn-kubernetes 2026/07/15 12:01:49 11:58:30 INFO Pod/model-registry-operator-controller-manager-5797446674-csvbn: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:afac48e4dd11212fb851dbacae4ce6ef1266e6ad29cfd83b0c22ed6eb21842f5" already present on machine 2026/07/15 12:01:49 11:58:29 INFO Pod/model-registry-operator-controller-manager-5797446674-csvbn: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-5797446674-csvbn to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 11:58:29 INFO Pod/odh-model-controller-bc67559db-ch8jk: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bc67559db-ch8jk to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:01:49 11:58:29 INFO Pod/odh-model-controller-bc67559db-ch8jk: AddedInterface - Add eth0 [10.129.1.145/23] from ovn-kubernetes 2026/07/15 12:01:49 11:58:29 INFO ReplicaSet/model-registry-operator-controller-manager-5797446674: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-5797446674-csvbn 2026/07/15 12:01:49 11:58:29 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-5797446674 from 0 to 1 2026/07/15 12:01:49 11:58:29 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bc67559db from 0 to 1 2026/07/15 12:01:49 11:58:29 INFO Pod/odh-model-controller-bc67559db-ch8jk: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 12:01:49 11:58:29 WARN Pod/model-registry-operator-controller-manager-5797446674-csvbn: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/15 12:01:49 11:58:29 INFO ReplicaSet/odh-model-controller-bc67559db: SuccessfulCreate - Created pod: odh-model-controller-bc67559db-ch8jk 2026/07/15 12:01:49 === RESOURCE QUOTAS === 2026/07/15 12:01:49 No resource quotas found in monitored namespaces 2026/07/15 12:01:49 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 12:01:49 Evidence: event Pod/model-registry-operator-controller-manager-5797446674-csvbn: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/model-registry-operator-controller-manager-5797446674-csvbn: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"model-registry-operator-webhook-service\" not found"],"confidence":"medium"} 2026/07/15 12:01:49 Diagnostics complete for failed test: group 2 controller_test.go:752: Stopping: group 2 test failed. 2026/07/15 12:01:49 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 12:01:49 Circuit breaker: infrastructure/unknown failure recorded (2/3 consecutive) 2026/07/15 12:05:28 Circuit breaker: success recorded, resetting failure counter (was 2) --- FAIL: TestOdhOperator/components (1327.99s) === RUN TestOdhOperator 2026/07/15 11:46:52 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/15 12:09:11 TEST FAILURE DETECTED: components 2026/07/15 12:09:11 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/15 12:09:11 === CLUSTER STATE === 2026/07/15 12:09:11 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/15 12:09:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:09:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:09:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:09:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:09:11 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/15 12:09:11 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:09:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:09:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:09:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:09:11 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/15 12:09:11 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/15 12:09:11 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/15 12:09:11 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/15 12:09:11 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/15 12:09:11 Condition Ready: True - kubelet is posting ready status 2026/07/15 12:09:11 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/15 12:09:11 === DEPLOYMENTS === 2026/07/15 12:09:11 Namespace: redhat-ods-applications 2026/07/15 12:09:11 Deployment dashboard-redirect: 2/2 (READY) 2026/07/15 12:09:11 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/15 12:09:11 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/15 12:09:11 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/15 12:09:11 Deployment maas-postgres: 1/1 (READY) 2026/07/15 12:09:11 Deployment model-serving-api: 1/1 (READY) 2026/07/15 12:09:11 Deployment odh-model-controller: 1/1 (READY) 2026/07/15 12:09:11 Namespace: kube-system 2026/07/15 12:09:11 === PODS === 2026/07/15 12:09:11 Namespace redhat-ods-applications: no problematic pods found 2026/07/15 12:09:11 Namespace kube-system: no problematic pods found 2026/07/15 12:09:11 === OPERATOR STATUS === 2026/07/15 12:09:11 Operator deployment: 3/3 ready 2026/07/15 12:09:11 === DSCI STATUS === 2026/07/15 12:09:11 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/15 12:09:11 === DSC STATUS === 2026/07/15 12:09:11 DSC default-dsc: 2026/07/15 12:09:11 Ready: True - 2026/07/15 12:09:11 ProvisioningSucceeded: True - 2026/07/15 12:09:11 ComponentsReady: True - 2026/07/15 12:09:11 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 DashboardReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 KserveLLMInferenceServiceDependencies: True - 2026/07/15 12:09:11 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/15 12:09:11 KserveReady: True - 2026/07/15 12:09:11 KueueReady: True - 2026/07/15 12:09:11 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 ModelControllerReady: True - 2026/07/15 12:09:11 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 OGXReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 RayReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 TrainerReady: True - 2026/07/15 12:09:11 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/15 12:09:11 === RECENT EVENTS === 2026/07/15 12:09:11 12:08:52 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-m692x_b9476e67-b490-4818-81c5-279dc884bb5c became leader 2026/07/15 12:09:11 12:08:52 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-x59cn_fbfaf39e-ee4e-41a5-9a5f-6437490d8608 became leader 2026/07/15 12:09:11 12:08:39 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-kj9hx_51f885e9-a306-4aa3-8948-154187bb466a became leader 2026/07/15 12:09:11 12:08:36 INFO Pod/kserve-controller-manager-7747874bf-x59cn: Started - Started container manager 2026/07/15 12:09:11 12:08:36 INFO Pod/kserve-controller-manager-7747874bf-x59cn: Created - Created container: manager 2026/07/15 12:09:11 12:08:36 INFO Pod/kserve-controller-manager-7747874bf-x59cn: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 2.157s (2.157s including waiting). Image size: 237680312 bytes. 2026/07/15 12:09:11 12:08:35 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-m692x: Started - Started container manager 2026/07/15 12:09:11 12:08:35 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-m692x: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 470ms (470ms including waiting). Image size: 235181729 bytes. 2026/07/15 12:09:11 12:08:35 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-m692x: Created - Created container: manager 2026/07/15 12:09:11 12:08:34 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-m692x: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-m692x to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:08:34 INFO Pod/kserve-controller-manager-7747874bf-x59cn: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-x59cn to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:08:34 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 12:09:11 12:08:34 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-m692x: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 12:09:11 12:08:34 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-m692x: AddedInterface - Add eth0 [10.129.1.181/23] from ovn-kubernetes 2026/07/15 12:09:11 12:08:34 INFO Pod/kserve-controller-manager-7747874bf-x59cn: AddedInterface - Add eth0 [10.129.1.180/23] from ovn-kubernetes 2026/07/15 12:09:11 12:08:34 INFO Pod/kserve-controller-manager-7747874bf-x59cn: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 12:09:11 12:08:34 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-m692x 2026/07/15 12:09:11 12:08:33 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-x59cn 2026/07/15 12:09:11 12:08:33 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 12:09:11 12:08:23 INFO Pod/model-serving-api-67c44bccdf-bcdbv: Created - Created container: server 2026/07/15 12:09:11 12:08:23 INFO Pod/model-serving-api-67c44bccdf-bcdbv: AddedInterface - Add eth0 [10.129.1.179/23] from ovn-kubernetes 2026/07/15 12:09:11 12:08:23 INFO Pod/odh-model-controller-5b978c765f-kj9hx: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 995ms (995ms including waiting). Image size: 238719886 bytes. 2026/07/15 12:09:11 12:08:23 INFO Pod/odh-model-controller-5b978c765f-kj9hx: Created - Created container: manager 2026/07/15 12:09:11 12:08:23 INFO Pod/model-serving-api-67c44bccdf-bcdbv: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 12:09:11 12:08:23 INFO Pod/odh-model-controller-5b978c765f-kj9hx: Started - Started container manager 2026/07/15 12:09:11 12:08:23 INFO Pod/model-serving-api-67c44bccdf-bcdbv: Started - Started container server 2026/07/15 12:09:11 12:08:22 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:08:22 INFO Pod/model-serving-api-67c44bccdf-bcdbv: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-bcdbv to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:08:22 INFO Pod/odh-model-controller-5b978c765f-kj9hx: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 12:09:11 12:08:22 INFO Pod/odh-model-controller-5b978c765f-kj9hx: AddedInterface - Add eth0 [10.129.1.178/23] from ovn-kubernetes 2026/07/15 12:09:11 12:08:22 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 12:09:11 12:08:22 WARN Pod/model-serving-api-67c44bccdf-bcdbv: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/15 12:09:11 12:08:22 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-bcdbv 2026/07/15 12:09:11 12:08:22 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 12:09:11 12:08:21 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:08:21 INFO Pod/odh-model-controller-5b978c765f-kj9hx: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-kj9hx to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:08:21 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-kj9hx 2026/07/15 12:09:11 12:08:21 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 12:09:11 12:08:21 WARN Pod/odh-model-controller-5b978c765f-kj9hx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 12:09:11 12:08:11 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:08:11 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: Killing - Stopping container manager 2026/07/15 12:09:11 12:08:11 INFO Pod/odh-model-controller-5b978c765f-xcvsz: Killing - Stopping container manager 2026/07/15 12:09:11 12:08:11 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: Killing - Stopping container manager 2026/07/15 12:09:11 12:08:11 INFO Pod/model-serving-api-67c44bccdf-kqcxj: Killing - Stopping container server 2026/07/15 12:09:11 12:07:46 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-gtxmr_de473432-a346-4558-a2bf-8a612023f455 became leader 2026/07/15 12:09:11 12:07:44 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-wt2zb_77bb39ea-807c-4949-8859-b9af35563892 became leader 2026/07/15 12:09:11 12:07:32 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-xcvsz_a5c3ff52-c223-45cd-9d7f-10f22de9c69c became leader 2026/07/15 12:09:11 12:07:27 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 1.497s (1.497s including waiting). Image size: 237680312 bytes. 2026/07/15 12:09:11 12:07:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: Created - Created container: manager 2026/07/15 12:09:11 12:07:27 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: Started - Started container manager 2026/07/15 12:09:11 12:07:27 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: Created - Created container: manager 2026/07/15 12:09:11 12:07:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 610ms (610ms including waiting). Image size: 235181729 bytes. 2026/07/15 12:09:11 12:07:27 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: Started - Started container manager 2026/07/15 12:09:11 12:07:26 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: AddedInterface - Add eth0 [10.129.1.177/23] from ovn-kubernetes 2026/07/15 12:09:11 12:07:26 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/15 12:09:11 12:07:26 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: AddedInterface - Add eth0 [10.129.1.176/23] from ovn-kubernetes 2026/07/15 12:09:11 12:07:26 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 12:09:11 12:07:26 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 12:09:11 12:07:25 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-gtxmr: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-gtxmr to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:07:25 INFO Pod/kserve-controller-manager-7747874bf-wt2zb: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-wt2zb to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:07:25 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 12:09:11 12:07:25 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-wt2zb 2026/07/15 12:09:11 12:07:25 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 12:09:11 12:07:25 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-gtxmr 2026/07/15 12:09:11 12:07:23 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-7dbc595499-kx6xm_22132a0b-0ee5-4d45-8aa1-415e659379f6 became leader 2026/07/15 12:09:11 12:07:14 INFO Pod/odh-model-controller-5b978c765f-xcvsz: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 2.234s (2.234s including waiting). Image size: 238719886 bytes. 2026/07/15 12:09:11 12:07:14 INFO Pod/odh-model-controller-5b978c765f-xcvsz: Created - Created container: manager 2026/07/15 12:09:11 12:07:14 INFO Pod/odh-model-controller-5b978c765f-xcvsz: Started - Started container manager 2026/07/15 12:09:11 12:07:12 INFO Pod/model-serving-api-67c44bccdf-kqcxj: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-kqcxj to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:07:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:07:12 INFO Pod/model-serving-api-67c44bccdf-kqcxj: Created - Created container: server 2026/07/15 12:09:11 12:07:12 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-kqcxj 2026/07/15 12:09:11 12:07:12 INFO Pod/model-serving-api-67c44bccdf-kqcxj: AddedInterface - Add eth0 [10.129.1.175/23] from ovn-kubernetes 2026/07/15 12:09:11 12:07:12 INFO Pod/model-serving-api-67c44bccdf-kqcxj: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 12:09:11 12:07:12 INFO Pod/model-serving-api-67c44bccdf-kqcxj: Started - Started container server 2026/07/15 12:09:11 12:07:11 INFO Pod/odh-model-controller-5b978c765f-xcvsz: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-xcvsz to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:07:11 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:07:11 WARN ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: FailedCreate - Error creating: pods "llmisvc-controller-manager-78dcbbdc6c-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/15 12:09:11 12:07:11 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-xcvsz 2026/07/15 12:09:11 12:07:11 INFO Pod/odh-model-controller-5b978c765f-xcvsz: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 12:09:11 12:07:11 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 12:09:11 12:07:11 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 12:09:11 12:07:11 WARN ReplicaSet/kserve-controller-manager-7747874bf: FailedCreate - Error creating: pods "kserve-controller-manager-7747874bf-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/15 12:09:11 12:07:11 INFO Pod/odh-model-controller-5b978c765f-xcvsz: AddedInterface - Add eth0 [10.129.1.174/23] from ovn-kubernetes 2026/07/15 12:09:11 12:07:11 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 12:09:11 12:07:11 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 12:09:11 12:07:06 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-kx6xm: AddedInterface - Add eth0 [10.129.1.173/23] from ovn-kubernetes 2026/07/15 12:09:11 12:07:06 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-kx6xm: Started - Started container manager 2026/07/15 12:09:11 12:07:06 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-kx6xm: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:595f456a6b3900dd112fa04416a2d91f448f8d68031b0c2de0d1dcb5fed28aa1" already present on machine 2026/07/15 12:09:11 12:07:06 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-kx6xm: Created - Created container: manager 2026/07/15 12:09:11 12:07:05 INFO Pod/kubeflow-trainer-controller-manager-7dbc595499-kx6xm: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-7dbc595499-kx6xm to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:07:05 WARN Service/kubeflow-trainer-controller-manager: ClusterIPNotAllocated - Cluster IP [IPv4]: 172.30.140.71 is not allocated; repairing 2026/07/15 12:09:11 12:07:05 INFO ReplicaSet/kubeflow-trainer-controller-manager-7dbc595499: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-7dbc595499-kx6xm 2026/07/15 12:09:11 12:07:05 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-7dbc595499 from 0 to 1 2026/07/15 12:09:11 12:07:01 INFO Pod/model-serving-api-67c44bccdf-2pf9b: Killing - Stopping container server 2026/07/15 12:09:11 12:07:01 INFO Pod/odh-model-controller-5b978c765f-wbxcx: Killing - Stopping container manager 2026/07/15 12:09:11 12:07:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:07:00 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: Killing - Stopping container manager 2026/07/15 12:09:11 12:07:00 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: Killing - Stopping container manager 2026/07/15 12:09:11 12:06:50 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:06:50 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:06:50 INFO Pod/maas-controller-85c784d7b6-7t4sh: Killing - Stopping container manager 2026/07/15 12:09:11 12:06:50 INFO Pod/maas-api-5db58d77cb-rw2x7: Killing - Stopping container maas-api 2026/07/15 12:09:11 12:06:19 INFO Pod/maas-api-5db58d77cb-rw2x7: Created - Created container: maas-api 2026/07/15 12:09:11 12:06:19 INFO Pod/maas-api-5db58d77cb-rw2x7: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" in 1.889s (1.889s including waiting). Image size: 174970750 bytes. 2026/07/15 12:09:11 12:06:19 INFO Pod/maas-api-5db58d77cb-rw2x7: Started - Started container maas-api 2026/07/15 12:09:11 12:06:17 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:06:17 WARN Pod/maas-api-5db58d77cb-rw2x7: FailedMount - MountVolume.SetUp failed for volume "maas-api-tls" : secret "maas-api-serving-cert" not found 2026/07/15 12:09:11 12:06:17 INFO Pod/maas-api-5db58d77cb-rw2x7: AddedInterface - Add eth0 [10.129.1.171/23] from ovn-kubernetes 2026/07/15 12:09:11 12:06:17 INFO Pod/maas-api-5db58d77cb-rw2x7: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:435d0a5a849ac8f7c05b7532b53158b29b2bead77295f2da203311777a0f0670" 2026/07/15 12:09:11 12:06:16 INFO Pod/maas-api-5db58d77cb-rw2x7: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5db58d77cb-rw2x7 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:06:16 WARN ReplicaSet/maas-api-5db58d77cb: FailedCreate - Error creating: pods "maas-api-5db58d77cb-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/15 12:09:11 12:06:16 INFO ReplicaSet/maas-api-5db58d77cb: SuccessfulCreate - Created pod: maas-api-5db58d77cb-rw2x7 2026/07/15 12:09:11 12:06:16 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5db58d77cb from 0 to 1 2026/07/15 12:09:11 12:06:02 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-7747874bf-8b9fb_03ccfcb7-0030-4502-94b9-49ce67db9b56 became leader 2026/07/15 12:09:11 12:06:01 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-78dcbbdc6c-588dv_cc7bb562-2881-4aeb-9084-a15dbb703304 became leader 2026/07/15 12:09:11 12:05:49 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-5b978c765f-wbxcx_c79bce70-a1ff-476b-8830-d5e00e887954 became leader 2026/07/15 12:09:11 12:05:46 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-85c784d7b6-7t4sh_7a1428a1-405e-406f-b00d-c478c5d09152 became leader 2026/07/15 12:09:11 12:05:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: Created - Created container: manager 2026/07/15 12:09:11 12:05:44 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: Created - Created container: manager 2026/07/15 12:09:11 12:05:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: Started - Started container manager 2026/07/15 12:09:11 12:05:44 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: Started - Started container manager 2026/07/15 12:09:11 12:05:44 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" in 633ms (633ms including waiting). Image size: 235181729 bytes. 2026/07/15 12:09:11 12:05:44 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" in 1.436s (1.436s including waiting). Image size: 237680312 bytes. 2026/07/15 12:09:11 12:05:43 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:4449e25f3216e819a156eb03bd39ea85e33609d803a8bd171b4de43a94f73ae7" 2026/07/15 12:09:11 12:05:43 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: AddedInterface - Add eth0 [10.129.1.169/23] from ovn-kubernetes 2026/07/15 12:09:11 12:05:43 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:7d3cc2a6f839315975db89bcf861d3cb332a6b23b6878fea1ad1dc4f62a52840" 2026/07/15 12:09:11 12:05:43 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: AddedInterface - Add eth0 [10.129.1.170/23] from ovn-kubernetes 2026/07/15 12:09:11 12:05:42 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-78dcbbdc6c-588dv to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:05:42 INFO Pod/kserve-controller-manager-7747874bf-8b9fb: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-7747874bf-8b9fb to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:05:42 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-78dcbbdc6c from 0 to 1 2026/07/15 12:09:11 12:05:42 WARN Pod/llmisvc-controller-manager-78dcbbdc6c-588dv: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/15 12:09:11 12:05:42 INFO ReplicaSet/kserve-controller-manager-7747874bf: SuccessfulCreate - Created pod: kserve-controller-manager-7747874bf-8b9fb 2026/07/15 12:09:11 12:05:42 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-7747874bf from 0 to 1 2026/07/15 12:09:11 12:05:42 INFO ReplicaSet/llmisvc-controller-manager-78dcbbdc6c: SuccessfulCreate - Created pod: llmisvc-controller-manager-78dcbbdc6c-588dv 2026/07/15 12:09:11 12:05:32 INFO Pod/model-serving-api-67c44bccdf-2pf9b: Started - Started container server 2026/07/15 12:09:11 12:05:32 INFO Pod/model-serving-api-67c44bccdf-2pf9b: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:226370d82da697958a0cd64b5cfa82f290ae1bb701e74b72739fcac0b38a834d" already present on machine 2026/07/15 12:09:11 12:05:32 INFO Pod/model-serving-api-67c44bccdf-2pf9b: AddedInterface - Add eth0 [10.129.1.168/23] from ovn-kubernetes 2026/07/15 12:09:11 12:05:32 INFO Pod/model-serving-api-67c44bccdf-2pf9b: Created - Created container: server 2026/07/15 12:09:11 12:05:31 INFO Pod/model-serving-api-67c44bccdf-2pf9b: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-67c44bccdf-2pf9b to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:05:31 INFO Pod/odh-model-controller-5b978c765f-wbxcx: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" in 632ms (632ms including waiting). Image size: 238719886 bytes. 2026/07/15 12:09:11 12:05:31 INFO Pod/odh-model-controller-5b978c765f-wbxcx: Created - Created container: manager 2026/07/15 12:09:11 12:05:31 INFO Pod/odh-model-controller-5b978c765f-wbxcx: Started - Started container manager 2026/07/15 12:09:11 12:05:31 WARN ReplicaSet/model-serving-api-67c44bccdf: FailedCreate - Error creating: pods "model-serving-api-67c44bccdf-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/15 12:09:11 12:05:31 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-67c44bccdf from 0 to 1 2026/07/15 12:09:11 12:05:31 INFO ReplicaSet/model-serving-api-67c44bccdf: SuccessfulCreate - Created pod: model-serving-api-67c44bccdf-2pf9b 2026/07/15 12:09:11 12:05:30 INFO Pod/odh-model-controller-5b978c765f-wbxcx: AddedInterface - Add eth0 [10.129.1.167/23] from ovn-kubernetes 2026/07/15 12:09:11 12:05:30 INFO Pod/maas-controller-85c784d7b6-7t4sh: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" 2026/07/15 12:09:11 12:05:30 INFO Pod/maas-controller-85c784d7b6-7t4sh: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:5effee51faa067115d737a4aa57473635576fb8232b4ea3cc39f76c0146abf65" in 494ms (494ms including waiting). Image size: 212346876 bytes. 2026/07/15 12:09:11 12:05:30 INFO Pod/odh-model-controller-5b978c765f-wbxcx: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b644dc155475dc76d7790e1b4d04dc7e86228b02650bfc38ac48292b7a9047a0" 2026/07/15 12:09:11 12:05:30 INFO Pod/maas-controller-85c784d7b6-7t4sh: Started - Started container manager 2026/07/15 12:09:11 12:05:30 INFO Pod/maas-controller-85c784d7b6-7t4sh: Created - Created container: manager 2026/07/15 12:09:11 12:05:29 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/15 12:09:11 12:05:29 INFO Pod/odh-model-controller-5b978c765f-wbxcx: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-5b978c765f-wbxcx to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:05:29 INFO Pod/maas-controller-85c784d7b6-7t4sh: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-85c784d7b6-7t4sh to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:05:29 WARN Pod/odh-model-controller-5b978c765f-wbxcx: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/15 12:09:11 12:05:29 INFO ReplicaSet/odh-model-controller-5b978c765f: SuccessfulCreate - Created pod: odh-model-controller-5b978c765f-wbxcx 2026/07/15 12:09:11 12:05:29 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-85c784d7b6 from 0 to 1 2026/07/15 12:09:11 12:05:29 INFO ReplicaSet/maas-controller-85c784d7b6: SuccessfulCreate - Created pod: maas-controller-85c784d7b6-7t4sh 2026/07/15 12:09:11 12:05:29 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-5b978c765f from 0 to 1 2026/07/15 12:09:11 12:05:29 INFO Pod/maas-controller-85c784d7b6-7t4sh: AddedInterface - Add eth0 [10.129.1.166/23] from ovn-kubernetes 2026/07/15 12:09:11 12:05:18 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh: Killing - Stopping container manager 2026/07/15 12:09:11 12:05:10 INFO Pod/maas-api-5db58d77cb-5zmrb: Killing - Stopping container maas-api 2026/07/15 12:09:11 12:05:10 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-bc99cf85b-cqjdh_542c5980-0211-4a5b-9ba0-2d20a43a8ef4 became leader 2026/07/15 12:09:11 12:05:10 INFO Pod/maas-controller-85c784d7b6-p67b4: Killing - Stopping container manager 2026/07/15 12:09:11 12:05:09 INFO Pod/llmisvc-controller-manager-78dcbbdc6c-g7jgd: Killing - Stopping container manager 2026/07/15 12:09:11 12:05:09 INFO Pod/model-serving-api-67c44bccdf-g44mm: Killing - Stopping container server 2026/07/15 12:09:11 12:05:09 INFO Pod/kserve-controller-manager-7747874bf-zvghz: Killing - Stopping container manager 2026/07/15 12:09:11 12:05:09 INFO Pod/odh-model-controller-5b978c765f-lssbp: Killing - Stopping container manager 2026/07/15 12:09:11 12:04:53 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh: AddedInterface - Add eth0 [10.129.1.165/23] from ovn-kubernetes 2026/07/15 12:09:11 12:04:53 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:780ba93a8a4bf718fe19ad7bb9e4dec7b0ae79d5eb6d3e1e5060c3dc2260a515" already present on machine 2026/07/15 12:09:11 12:04:53 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh: Created - Created container: manager 2026/07/15 12:09:11 12:04:53 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh: Started - Started container manager 2026/07/15 12:09:11 12:04:52 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-bc99cf85b-cqjdh to ip-10-0-1-45.us-east-2.compute.internal 2026/07/15 12:09:11 12:04:52 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-pqxcp: Killing - Stopping container manager 2026/07/15 12:09:11 12:04:52 INFO Pod/trustyai-service-operator-controller-manager-bc99cf85b-x8hcg: Killing - Stopping container manager 2026/07/15 12:09:11 12:04:52 INFO ReplicaSet/trustyai-service-operator-controller-manager-bc99cf85b: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-bc99cf85b-cqjdh 2026/07/15 12:09:11 12:04:52 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-bc99cf85b from 0 to 1 2026/07/15 12:09:11 === RESOURCE QUOTAS === 2026/07/15 12:09:11 No resource quotas found in monitored namespaces 2026/07/15 12:09:11 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/15 12:09:11 Evidence: event Pod/model-serving-api-67c44bccdf-bcdbv: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/model-serving-api-67c44bccdf-bcdbv: FailedMount - MountVolume.SetUp failed for volume \"tls-certs\" : secret \"model-serving-api-tls\" not found"],"confidence":"medium"} 2026/07/15 12:09:11 Diagnostics complete for failed test: components controller_test.go:483: Stopping: components test failed. 2026/07/15 12:09:11 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/15 12:09:11 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/15 12:09:12 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1340.47s)