=== RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-12T18:40:43Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com, certSecret=2r2qdb90ltvhjc43lp6iqc6aqb7bgjda-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners resource_applier_test.go:202: Timed out after 600.004s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-dsc' of kind 'DataScienceCluster' to match condition {.status.conditions[] | select(.type == "ModelsAsServiceReady") | .status == "True" []}. Expected <string>: map[apiVersion:datasciencecluster.opendatahub.io/v2 kind:DataScienceCluster metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"datasciencecluster.opendatahub.io/v2","kind":"DataScienceCluster","metadata":{"annotations":{},"name":"default-dsc"},"spec":{"components":{"aipipelines":{"managementState":"Managed"},"codeflare":{"managementState":"Managed"},"dashboard":{"managementState":"Managed"},"feastoperator":{"managementState":"Managed"},"kserve":{"defaultDeploymentMode":"RawDeployment","managementState":"Managed","serving":{"managementState":"Removed","name":"knative-serving"}},"kueue":{"managementState":"Removed"},"llamastackoperator":{"managementState":"Removed"},"mlflowoperator":{"managementState":"Managed"},"modelregistry":{"managementState":"Managed"},"ogx":{"managementState":"Removed"},"ray":{"managementState":"Managed"},"sparkoperator":{"managementState":"Removed"},"trainer":{"managementState":"Managed"},"trainingoperator":{"managementState":"Removed"},"trustyai":{"managementState":"Managed"},"workbenches":{"managementState":"Managed"}}}} ] creationTimestamp:2026-07-10T06:04:35Z generation:125 managedFields:[map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aigateway:map[] f:aipipelines:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Pipelines"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Feast"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kserve:map[f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-router"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[] f:registriesNamespace:map[] f:releases:map[k:{"name":"Kubeflow Model Registry"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Open Data Hub Model Registry Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ogx:map[f:managementState:map[] f:releases:map[k:{"name":"OGX Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"OGX"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ray:map[f:managementState:map[] f:releases:map[k:{"name":"KubeRay"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Trainer"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainingoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Training Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Notebook Controller"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]] f:workbenchNamespace:map[]]] f:release:map[f:name:map[] f:version:map[]]]] manager:modules operation:Apply subresource:status time:2026-07-12T18:40:45Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aigateway:map[] f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-router"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]] f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:datasciencecluster operation:Apply subresource:status time:2026-07-12T18:41:33Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:spec:map[.:map[] f:components:map[.:map[] f:aipipelines:map[] f:dashboard:map[] f:feastoperator:map[] f:kserve:map[.:map[] f:nim:map[.:map[] f:airGapped:map[] f:managementState:map[]] f:rawDeploymentServiceConfig:map[]] f:kueue:map[.:map[] f:defaultClusterQueueName:map[] f:defaultLocalQueueName:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[] f:modelregistry:map[.:map[] f:registriesNamespace:map[]] f:ogx:map[] f:ray:map[] f:sparkoperator:map[] f:trainer:map[] f:trainingoperator:map[] f:trustyai:map[.:map[] f:mcpGuardrailsMode:map[]] f:workbenches:map[.:map[] f:workbenchNamespace:map[]]]]] manager:kubectl-client-side-apply operation:Update time:2026-07-10T06:04:35Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:kserve:map[f:modelsAsService:map[.:map[] f:managementState:map[]]] f:llamastackoperator:map[f:managementState:map[]]]]] manager:kubectl-patch operation:Update time:2026-07-12T17:41:15Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:aigateway:map[.:map[] f:managementState:map[]] f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:wva:map[.:map[] f:managementState:map[]]] f:kueue:map[f:managementState:map[]] f:mlflowoperator:map[f:managementState:map[]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]]]] manager:e2e-tests operation:Update time:2026-07-12T18:40:43Z]] name:default-dsc resourceVersion:26639431 uid:c2bc1b4c-2489-4024-bd90-fc56fa75590a] spec:map[components:map[aigateway:map[managementState:Removed] aipipelines:map[managementState:Removed] dashboard:map[managementState:Removed] feastoperator:map[managementState:Removed] kserve:map[managementState:Managed modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] kueue:map[defaultClusterQueueName:default defaultLocalQueueName:default managementState:Removed] llamastackoperator:map[managementState:Removed] mlflowoperator:map[managementState:Removed] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries] ogx:map[managementState:Removed] ray:map[managementState:Removed] sparkoperator:map[managementState:Removed] trainer:map[managementState:Removed] trainingoperator:map[managementState:Removed] trustyai:map[managementState:Removed mcpGuardrailsMode:false] workbenches:map[managementState:Removed workbenchNamespace:rhods-notebooks]]] status:map[components:map[aigateway:map[] aipipelines:map[managementState:Removed releases:[map[name:Kubeflow Pipelines repoUrl:https://github.com/kubeflow/pipelines version:2.16.0]]] dashboard:map[managementState:Removed] feastoperator:map[managementState:Removed releases:[map[name:Feast repoUrl:https://github.com/feast-dev/feast version:0.64.0]]] kserve:map[managementState:Managed releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.19.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.21.0] map[name:llm-d-router repoUrl:https://github.com/llm-d/llm-d-router version:v0.9.0] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.8.0]]] kueue:map[managementState:Removed] llamastackoperator:map[] mlflowoperator:map[managementState:Removed releases:[map[name:MLflow repoUrl:https://github.com/mlflow/mlflow version:v3.13.0]]] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries releases:[map[name:Kubeflow Model Registry repoUrl:https://github.com/kubeflow/model-registry version:latest] map[name:Open Data Hub Model Registry Operator repoUrl:https://github.com/opendatahub-io/model-registry-operator version:latest]]] ogx:map[managementState:Removed releases:[map[name:OGX repoUrl:https://github.com/opendatahub-io/ogx-distribution version:v1.1.2] map[name:OGX Operator repoUrl:https://github.com/opendatahub-io/ogx-k8s-operator version:v0.11.0]]] ray:map[managementState:Removed releases:[map[name:KubeRay repoUrl:https://github.com/ray-project/kuberay version:1.4.2]]] sparkoperator:map[managementState:Removed releases:[map[name:Spark Operator repoUrl:https://github.com/opendatahub-io/spark-operator version:v2.4.0] map[name:Spark repoUrl:https://github.com/apache/spark version:4.0.1]]] trainer:map[managementState:Removed releases:[map[name:Kubeflow Trainer repoUrl:https://github.com/kubeflow/trainer version:2.1.0]]] trainingoperator:map[managementState:Removed releases:[map[name:Kubeflow Training Operator repoUrl:https://github.com/kubeflow/trainer version:1.9.0]]] trustyai:map[managementState:Removed releases:[map[name:TrustyAI operator repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI service repoUrl:https://github.com/trustyai-explainability/trustyai-explainability version:v0.28.0] map[name:TrustyAI LMEval driver repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI LMEval job repoUrl:https://github.com/EleutherAI/lm-evaluation-harness version:v0.4.8] map[name:TrustyAI Guardrails orchestrator repoUrl:https://github.com/foundation-model-stack/fms-guardrails-orchestrator version:0.9.4] map[name:TrustyAI builtin detectors repoUrl:https://github.com/trustyai-explainability/guardrails-regex-detector version:v0.2.0] map[name:TrustyAI sidecar gateway repoUrl:https://github.com/trustyai-explainability/vllm-orchestrator-gateway version:v0.2.1] map[name:EvalHub repoUrl:https://github.com/eval-hub/eval-hub version:v0.3.0]]] workbenches:map[managementState:Removed releases:[map[name:Kubeflow Notebook Controller repoUrl:https://github.com/kubeflow/kubeflow version:1.10.0]] workbenchNamespace:rhods-notebooks]] conditions:[map[lastTransitionTime:2026-07-12T18:41:33Z message:Some components are not ready: modelsasservice reason:NotReady status:False type:Ready] map[lastTransitionTime:2026-07-12T18:40:45Z observedGeneration:125 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-12T18:41:33Z message:Some components are not ready: modelsasservice reason:NotReady status:False type:ComponentsReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:AIPipelinesReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:DashboardReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:FeastOperatorReady] map[lastTransitionTime:2026-07-12T18:40:59Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-12T18:40:59Z status:True type:KserveLLMInferenceServiceWideEPDependencies] map[lastTransitionTime:2026-07-12T18:41:33Z status:True type:KserveReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:KueueReady] map[lastTransitionTime:2026-07-12T18:40:51Z status:True type:LLM-D-WVADependencies] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:MLflowOperatorReady] map[lastTransitionTime:2026-07-12T18:40:43Z message:maas-default-gateway is missing required annotations: security.opendatahub.io/authorino-tls-bootstrap="true" reason:MaaSPrerequisitesNotMet severity:Info status:False type:MaaSPrerequisitesAvailable] map[lastTransitionTime:2026-07-12T18:41:33Z status:True type:ModelControllerReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:ModelRegistryReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Tenant CR not available yet reason:NotReady status:False type:ModelsAsServiceReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:OGXReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:RayReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:SparkOperatorReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainerReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainingOperatorReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrustyAIReady] map[lastTransitionTime:2026-07-12T18:41:33Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:WorkbenchesReady]] observedGeneration:124 phase:Not Ready release:map[name:OpenShift AI Self-Managed version:3.5.0-ea.2]]] to match expression <string>: .status.conditions[] | select(.type == "ModelsAsServiceReady") | .status == "True" --- FAIL: TestOdhOperator/components/group_4/modelsasservice (650.93s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/12 18:51:34 TEST FAILURE DETECTED: group 4 2026/07/12 18:51:34 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/12 18:51:34 === CLUSTER STATE === 2026/07/12 18:51:34 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:51:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:51:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:51:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:51:34 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:51:34 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/12 18:51:34 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/12 18:51:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:51:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:51:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:51:34 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:51:34 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/12 18:51:34 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/12 18:51:34 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:51:34 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:51:34 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:51:34 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:51:34 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/12 18:51:34 === DEPLOYMENTS === 2026/07/12 18:51:34 Namespace: redhat-ods-applications 2026/07/12 18:51:34 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/12 18:51:34 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/12 18:51:34 Deployment maas-postgres: 1/1 (READY) 2026/07/12 18:51:34 Deployment model-serving-api: 1/1 (READY) 2026/07/12 18:51:34 Deployment odh-model-controller: 1/1 (READY) 2026/07/12 18:51:34 Namespace: kube-system 2026/07/12 18:51:34 === PODS === 2026/07/12 18:51:34 Namespace: redhat-ods-applications 2026/07/12 18:51:34 Pod maas-api-key-cleanup-29728335-zqlrm: Succeeded 2026/07/12 18:51:34 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/12 18:51:34 Namespace kube-system: no problematic pods found 2026/07/12 18:51:34 === LOGS (CURRENT) for container cleanup === 2026/07/12 18:51:34 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/12 18:51:34 === END LOGS === 2026/07/12 18:51:34 === OPERATOR STATUS === 2026/07/12 18:51:34 Operator deployment: 3/3 ready 2026/07/12 18:51:34 === DSCI STATUS === 2026/07/12 18:51:34 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 18:51:34 === DSC STATUS === 2026/07/12 18:51:34 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; MaaSPrerequisitesAvailable=False; ModelsAsServiceReady=False 2026/07/12 18:51:34 === RECENT EVENTS === 2026/07/12 18:51:34 18:50:58 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:51:34 18:50:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:51:34 18:50:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:51:34 === RESOURCE QUOTAS === 2026/07/12 18:51:34 No resource quotas found in monitored namespaces 2026/07/12 18:51:34 Classification for group 4: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/12 18:51:34 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/12 18:51:34 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/12 18:51:34 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/12 18:51:34 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/12 18:52:45 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1589.44s) === RUN TestOdhOperator 2026/07/12 18:27:06 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/12 18:53:46 TEST FAILURE DETECTED: components 2026/07/12 18:53:46 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/12 18:53:47 === CLUSTER STATE === 2026/07/12 18:53:47 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:53:47 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:53:47 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:53:47 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:53:47 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/12 18:53:47 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/12 18:53:47 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:53:47 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:53:47 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:53:47 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:53:47 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/12 18:53:47 Node: ip-10-0-1-62.us-east-2.compute.internal 2026/07/12 18:53:47 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/12 18:53:47 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/12 18:53:47 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/12 18:53:47 Condition Ready: True - kubelet is posting ready status 2026/07/12 18:53:47 Resources: 15500m CPU, 63182896Ki memory allocatable (of 16000m CPU, 64333872Ki memory capacity) 2026/07/12 18:53:47 === DEPLOYMENTS === 2026/07/12 18:53:47 Namespace: redhat-ods-applications 2026/07/12 18:53:47 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/12 18:53:47 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/12 18:53:47 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/12 18:53:47 Deployment maas-postgres: 1/1 (READY) 2026/07/12 18:53:47 Deployment model-serving-api: 1/1 (READY) 2026/07/12 18:53:47 Deployment odh-model-controller: 1/1 (READY) 2026/07/12 18:53:47 Namespace: kube-system 2026/07/12 18:53:47 === PODS === 2026/07/12 18:53:47 Namespace: redhat-ods-applications 2026/07/12 18:53:47 Pod maas-api-key-cleanup-29728335-zqlrm: Succeeded 2026/07/12 18:53:47 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/12 18:53:47 Namespace kube-system: no problematic pods found 2026/07/12 18:53:47 === LOGS (CURRENT) for container cleanup === 2026/07/12 18:53:47 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/12 18:53:47 === END LOGS === 2026/07/12 18:53:47 === OPERATOR STATUS === 2026/07/12 18:53:47 Operator deployment: 3/3 ready 2026/07/12 18:53:47 === DSCI STATUS === 2026/07/12 18:53:47 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/12 18:53:47 === DSC STATUS === 2026/07/12 18:53:47 Failed to collect DSC data: default-dsc unhealthy conditions: Ready=False; ComponentsReady=False; MaaSPrerequisitesAvailable=False; ModelsAsServiceReady=False 2026/07/12 18:53:47 === RECENT EVENTS === 2026/07/12 18:53:47 18:53:30 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-tv4kv_f3265f75-f050-45c6-853c-8777246c629a became leader 2026/07/12 18:53:47 18:53:28 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-hvz7f_64151b91-5bfc-4311-8396-3a54a5d66bb4 became leader 2026/07/12 18:53:47 18:53:15 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-skpj5_ba894a03-5a47-4da4-af63-dc125b8057e8 became leader 2026/07/12 18:53:47 18:53:12 INFO Pod/kserve-controller-manager-dd9f6cf9-hvz7f: Created - Created container: manager 2026/07/12 18:53:47 18:53:12 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-tv4kv: Started - Started container manager 2026/07/12 18:53:47 18:53:12 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-tv4kv: Created - Created container: manager 2026/07/12 18:53:47 18:53:12 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-tv4kv: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 557ms (557ms including waiting). Image size: 236366599 bytes. 2026/07/12 18:53:47 18:53:12 INFO Pod/kserve-controller-manager-dd9f6cf9-hvz7f: Started - Started container manager 2026/07/12 18:53:47 18:53:11 INFO Pod/kserve-controller-manager-dd9f6cf9-hvz7f: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 396ms (396ms including waiting). Image size: 238435613 bytes. 2026/07/12 18:53:47 18:53:11 INFO Pod/kserve-controller-manager-dd9f6cf9-hvz7f: AddedInterface - Add eth0 [10.130.0.186/23] from ovn-kubernetes 2026/07/12 18:53:47 18:53:11 INFO Pod/kserve-controller-manager-dd9f6cf9-hvz7f: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 18:53:47 18:53:11 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-tv4kv: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 18:53:47 18:53:11 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-tv4kv: AddedInterface - Add eth0 [10.130.0.187/23] from ovn-kubernetes 2026/07/12 18:53:47 18:53:10 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-tv4kv: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-tv4kv to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:53:10 INFO Pod/kserve-controller-manager-dd9f6cf9-hvz7f: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-hvz7f to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:53:10 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-tv4kv 2026/07/12 18:53:47 18:53:10 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-hvz7f 2026/07/12 18:53:47 18:53:10 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/12 18:53:47 18:53:10 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/12 18:53:47 18:52:59 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:52:59 INFO Pod/model-serving-api-5679d6b4f-xpfqs: Started - Started container server 2026/07/12 18:53:47 18:52:59 INFO Pod/model-serving-api-5679d6b4f-xpfqs: Created - Created container: server 2026/07/12 18:53:47 18:52:59 INFO Pod/model-serving-api-5679d6b4f-xpfqs: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 18:53:47 18:52:59 INFO Pod/model-serving-api-5679d6b4f-xpfqs: AddedInterface - Add eth0 [10.130.0.185/23] from ovn-kubernetes 2026/07/12 18:53:47 18:52:58 INFO Pod/model-serving-api-5679d6b4f-xpfqs: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-xpfqs to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:52:58 INFO Pod/odh-model-controller-bd4c5cbcb-skpj5: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 459ms (459ms including waiting). Image size: 238959091 bytes. 2026/07/12 18:53:47 18:52:58 INFO Pod/odh-model-controller-bd4c5cbcb-skpj5: AddedInterface - Add eth0 [10.130.0.184/23] from ovn-kubernetes 2026/07/12 18:53:47 18:52:58 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/12 18:53:47 18:52:58 INFO Pod/odh-model-controller-bd4c5cbcb-skpj5: Started - Started container manager 2026/07/12 18:53:47 18:52:58 INFO Pod/odh-model-controller-bd4c5cbcb-skpj5: Created - Created container: manager 2026/07/12 18:53:47 18:52:58 INFO Pod/odh-model-controller-bd4c5cbcb-skpj5: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 18:53:47 18:52:58 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 18:53:47 18:52:58 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-xpfqs 2026/07/12 18:53:47 18:52:57 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:52:57 WARN Pod/odh-model-controller-bd4c5cbcb-skpj5: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 18:53:47 18:52:56 INFO Pod/odh-model-controller-bd4c5cbcb-skpj5: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-skpj5 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:52:56 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-skpj5 2026/07/12 18:53:47 18:52:56 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/12 18:53:47 18:52:47 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: Killing - Stopping container manager 2026/07/12 18:53:47 18:52:47 INFO Pod/model-serving-api-5679d6b4f-2c6cz: Killing - Stopping container server 2026/07/12 18:53:47 18:52:47 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: Killing - Stopping container manager 2026/07/12 18:53:47 18:52:46 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: Killing - Stopping container manager 2026/07/12 18:53:47 18:52:26 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-wrnkg_5916830c-3e33-401c-a323-6d1b76caf315 became leader 2026/07/12 18:53:47 18:52:24 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-f8ph9_732b2a38-8279-4b56-b82a-7d6feb7b1558 became leader 2026/07/12 18:53:47 18:52:09 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: Started - Started container manager 2026/07/12 18:53:47 18:52:09 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: Created - Created container: manager 2026/07/12 18:53:47 18:52:09 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 505ms (505ms including waiting). Image size: 236366599 bytes. 2026/07/12 18:53:47 18:52:08 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/12 18:53:47 18:52:08 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 586ms (586ms including waiting). Image size: 238435613 bytes. 2026/07/12 18:53:47 18:52:08 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: Created - Created container: manager 2026/07/12 18:53:47 18:52:08 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: Started - Started container manager 2026/07/12 18:53:47 18:52:08 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: AddedInterface - Add eth0 [10.130.0.183/23] from ovn-kubernetes 2026/07/12 18:53:47 18:52:07 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-wrnkg to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:52:07 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-f8ph9 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:52:07 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-f8ph9 2026/07/12 18:53:47 18:52:07 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-wrnkg: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/12 18:53:47 18:52:07 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: AddedInterface - Add eth0 [10.130.0.182/23] from ovn-kubernetes 2026/07/12 18:53:47 18:52:07 INFO Pod/kserve-controller-manager-dd9f6cf9-f8ph9: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/12 18:53:47 18:52:07 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/12 18:53:47 18:52:07 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/12 18:53:47 18:52:07 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-wrnkg 2026/07/12 18:53:47 18:52:04 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-jnllb_b71d15e3-6cff-42a1-be5c-e267a6078228 became leader 2026/07/12 18:53:47 18:51:55 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz_c96037c7-ca28-424d-bd7f-dadea2b5f44e became leader 2026/07/12 18:53:47 18:51:49 INFO Pod/model-serving-api-5679d6b4f-2c6cz: AddedInterface - Add eth0 [10.130.0.181/23] from ovn-kubernetes 2026/07/12 18:53:47 18:51:49 INFO Pod/model-serving-api-5679d6b4f-2c6cz: Started - Started container server 2026/07/12 18:53:47 18:51:49 INFO Pod/model-serving-api-5679d6b4f-2c6cz: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/12 18:53:47 18:51:49 INFO Pod/model-serving-api-5679d6b4f-2c6cz: Created - Created container: server 2026/07/12 18:53:47 18:51:48 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:51:48 INFO Pod/model-serving-api-5679d6b4f-2c6cz: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-2c6cz to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:51:48 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: Created - Created container: manager 2026/07/12 18:53:47 18:51:48 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-2c6cz 2026/07/12 18:53:47 18:51:48 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: Started - Started container manager 2026/07/12 18:53:47 18:51:48 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/12 18:53:47 18:51:48 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 477ms (477ms including waiting). Image size: 238959091 bytes. 2026/07/12 18:53:47 18:51:48 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/12 18:53:47 18:51:47 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/12 18:53:47 18:51:47 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: AddedInterface - Add eth0 [10.130.0.180/23] from ovn-kubernetes 2026/07/12 18:53:47 18:51:46 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:51:46 INFO Pod/odh-model-controller-bd4c5cbcb-jnllb: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-jnllb to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:51:46 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/12 18:53:47 18:51:46 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-jnllb 2026/07/12 18:53:47 18:51:46 WARN Pod/odh-model-controller-bd4c5cbcb-jnllb: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/12 18:53:47 18:51:40 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz: Started - Started container manager 2026/07/12 18:53:47 18:51:40 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz: Created - Created container: manager 2026/07/12 18:53:47 18:51:39 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz to ip-10-0-1-23.us-east-2.compute.internal 2026/07/12 18:53:47 18:51:39 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/12 18:53:47 18:51:39 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz 2026/07/12 18:53:47 18:51:39 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/12 18:53:47 18:51:39 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-nmbdz: AddedInterface - Add eth0 [10.130.0.179/23] from ovn-kubernetes 2026/07/12 18:53:47 18:51:36 INFO Pod/odh-model-controller-bd4c5cbcb-l5pnd: Killing - Stopping container manager 2026/07/12 18:53:47 18:51:36 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-fr98f: Killing - Stopping container manager 2026/07/12 18:53:47 18:51:36 INFO Pod/model-serving-api-5679d6b4f-vz2vb: Killing - Stopping container server 2026/07/12 18:53:47 18:51:36 INFO Pod/kserve-controller-manager-dd9f6cf9-qplv5: Killing - Stopping container manager 2026/07/12 18:53:47 18:51:35 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:50:58 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:50:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 18:50:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/12 18:53:47 === RESOURCE QUOTAS === 2026/07/12 18:53:47 No resource quotas found in monitored namespaces 2026/07/12 18:53:47 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/12 18:53:47 Evidence: event Pod/odh-model-controller-bd4c5cbcb-skpj5: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/odh-model-controller-bd4c5cbcb-skpj5: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"odh-model-controller-webhook-cert\" not found"],"confidence":"medium"} 2026/07/12 18:53:47 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/12 18:53:47 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/12 18:53:47 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/12 18:53:48 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1601.91s)