=== RUN TestOdhOperator/components/group_1/aigateway/Validate_component_enabled 2026-07-09T20:27:42Z INFO unknown field "spec.components.aigateway" 2026-07-09T20:27:42Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} 2026-07-09T20:27:42Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/aigateway/Validate_component_enabled (300.00s) === RUN TestOdhOperator/components/group_1/aigateway === PAUSE TestOdhOperator/components/group_1/aigateway === CONT TestOdhOperator/components/group_1/aigateway resource_applier_test.go:202: Timed out after 300.000s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-dsc' of kind 'DataScienceCluster' to match condition {.spec.components.aigateway.managementState == "Removed" []}. Expected <string>: map[apiVersion:datasciencecluster.opendatahub.io/v2 kind:DataScienceCluster metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"datasciencecluster.opendatahub.io/v2","kind":"DataScienceCluster","metadata":{"annotations":{},"name":"default-dsc"},"spec":{"components":{"aipipelines":{"managementState":"Removed"},"codeflare":{"managementState":"Removed"},"dashboard":{"managementState":"Managed"},"kserve":{"defaultDeploymentMode":"RawDeployment","managementState":"Managed","serving":{"managementState":"Removed","name":"knative-serving"}},"kueue":{"managementState":"Removed"},"llamastackoperator":{"managementState":"Removed"},"mlflowoperator":{"managementState":"Removed"},"modelregistry":{"managementState":"Removed"},"ogx":{"managementState":"Removed"},"ray":{"managementState":"Removed"},"sparkoperator":{"managementState":"Removed"},"trainer":{"managementState":"Removed"},"trainingoperator":{"managementState":"Removed"},"trustyai":{"managementState":"Removed"},"workbenches":{"managementState":"Removed"}}}} ] creationTimestamp:2026-06-29T13:43:26Z generation:2212 managedFields:[map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aipipelines:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Pipelines"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Feast"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[] f:registriesNamespace:map[] f:releases:map[k:{"name":"Kubeflow Model Registry"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Open Data Hub Model Registry Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ogx:map[f:managementState:map[] f:releases:map[k:{"name":"OGX Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"OGX"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:ray:map[f:managementState:map[] f:releases:map[k:{"name":"KubeRay"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:releases:map[k:{"name":"Kubeflow Trainer"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainingoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Training Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[] f:releases:map[k:{"name":"Kubeflow Notebook Controller"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]] f:workbenchNamespace:map[]]] f:observedGeneration:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:modules operation:Apply subresource:status time:2026-07-09T20:31:01Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:status:map[f:components:map[f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[f:managementState:map[]] f:kserve:map[f:managementState:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[f:managementState:map[] f:releases:map[k:{"name":"MLflow"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[] f:releases:map[k:{"name":"Spark Operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"Spark"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[] f:releases:map[k:{"name":"EvalHub"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI Guardrails orchestrator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval driver"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI LMEval job"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI builtin detectors"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI operator"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI service"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"TrustyAI sidecar gateway"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]] f:workbenches:map[f:managementState:map[]]] f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:release:map[f:name:map[] f:version:map[]]]] manager:datasciencecluster operation:Apply subresource:status time:2026-07-09T20:32:07Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:kubectl.kubernetes.io/last-applied-configuration:map[]]] f:spec:map[.:map[] f:components:map[.:map[] f:aipipelines:map[] f:dashboard:map[] f:kserve:map[.:map[] f:rawDeploymentServiceConfig:map[]] f:kueue:map[.:map[] f:defaultClusterQueueName:map[] f:defaultLocalQueueName:map[]] f:llamastackoperator:map[] f:mlflowoperator:map[] f:modelregistry:map[.:map[] f:registriesNamespace:map[]] f:ogx:map[] f:ray:map[] f:sparkoperator:map[] f:trainer:map[] f:trainingoperator:map[] f:trustyai:map[.:map[] f:mcpGuardrailsMode:map[]] f:workbenches:map[.:map[] f:workbenchNamespace:map[]]]]] manager:kubectl-client-side-apply operation:Update time:2026-06-29T13:43:26Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:kserve:map[f:modelsAsService:map[.:map[] f:managementState:map[]]] f:kueue:map[f:managementState:map[]] f:llamastackoperator:map[f:managementState:map[]] f:mlflowoperator:map[f:managementState:map[]] f:trustyai:map[f:managementState:map[]]]]] manager:kubectl-patch operation:Update time:2026-07-09T19:00:32Z] map[apiVersion:datasciencecluster.opendatahub.io/v2 fieldsType:FieldsV1 fieldsV1:map[f:spec:map[f:components:map[f:aipipelines:map[f:managementState:map[]] f:dashboard:map[f:managementState:map[]] f:feastoperator:map[.:map[] f:managementState:map[]] f:kserve:map[f:managementState:map[] f:wva:map[.:map[] f:managementState:map[]]] f:modelregistry:map[f:managementState:map[]] f:ogx:map[f:managementState:map[]] f:ray:map[f:managementState:map[]] f:sparkoperator:map[f:managementState:map[]] f:trainer:map[f:managementState:map[]] f:trainingoperator:map[f:managementState:map[]] f:workbenches:map[f:managementState:map[]]]]] manager:e2e-tests operation:Update time:2026-07-09T20:31:01Z]] name:default-dsc resourceVersion:386847312 uid:191cdb15-01aa-4da4-9798-e6b508adbf9f] spec:map[components:map[aipipelines:map[managementState:Removed] dashboard:map[managementState:Managed] feastoperator:map[managementState:Removed] kserve:map[managementState:Managed modelsAsService:map[managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] kueue:map[defaultClusterQueueName:default defaultLocalQueueName:default managementState:Unmanaged] llamastackoperator:map[managementState:Removed] mlflowoperator:map[managementState:Managed] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries] ogx:map[managementState:Removed] ray:map[managementState:Removed] sparkoperator:map[managementState:Managed] trainer:map[managementState:Removed] trainingoperator:map[managementState:Removed] trustyai:map[managementState:Managed mcpGuardrailsMode:false] workbenches:map[managementState:Removed workbenchNamespace:rhods-notebooks]]] status:map[components:map[aipipelines:map[managementState:Removed releases:[map[name:Kubeflow Pipelines repoUrl:https://github.com/kubeflow/pipelines version:2.16.0]]] dashboard:map[managementState:Managed] feastoperator:map[managementState:Removed releases:[map[name:Feast repoUrl:https://github.com/feast-dev/feast version:0.63.0]]] kserve:map[managementState:Managed releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]] kueue:map[managementState:Unmanaged] llamastackoperator:map[] mlflowoperator:map[managementState:Managed releases:[map[name:MLflow repoUrl:https://github.com/mlflow/mlflow version:v3.12.0]]] modelregistry:map[managementState:Removed registriesNamespace:rhoai-model-registries releases:[map[name:Kubeflow Model Registry repoUrl:https://github.com/kubeflow/model-registry version:latest] map[name:Open Data Hub Model Registry Operator repoUrl:https://github.com/opendatahub-io/model-registry-operator version:latest]]] ogx:map[managementState:Removed releases:[map[name:OGX repoUrl:https://github.com/opendatahub-io/ogx-distribution version:v1.0.2] map[name:OGX Operator repoUrl:https://github.com/opendatahub-io/ogx-k8s-operator version:v0.10.0]]] ray:map[managementState:Removed releases:[map[name:KubeRay repoUrl:https://github.com/ray-project/kuberay version:1.4.2]]] sparkoperator:map[managementState:Managed releases:[map[name:Spark Operator repoUrl:https://github.com/opendatahub-io/spark-operator version:v2.4.0] map[name:Spark repoUrl:https://github.com/apache/spark version:4.0.1]]] trainer:map[managementState:Removed releases:[map[name:Kubeflow Trainer repoUrl:https://github.com/kubeflow/trainer version:2.1.0]]] trainingoperator:map[managementState:Removed releases:[map[name:Kubeflow Training Operator repoUrl:https://github.com/kubeflow/trainer version:1.9.0]]] trustyai:map[managementState:Managed releases:[map[name:TrustyAI operator repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI service repoUrl:https://github.com/trustyai-explainability/trustyai-explainability version:v0.28.0] map[name:TrustyAI LMEval driver repoUrl:https://github.com/trustyai-explainability/trustyai-service-operator version:v1.37.0] map[name:TrustyAI LMEval job repoUrl:https://github.com/EleutherAI/lm-evaluation-harness version:v0.4.8] map[name:TrustyAI Guardrails orchestrator repoUrl:https://github.com/foundation-model-stack/fms-guardrails-orchestrator version:0.9.4] map[name:TrustyAI builtin detectors repoUrl:https://github.com/trustyai-explainability/guardrails-regex-detector version:v0.2.0] map[name:TrustyAI sidecar gateway repoUrl:https://github.com/trustyai-explainability/vllm-orchestrator-gateway version:v0.2.1] map[name:EvalHub repoUrl:https://github.com/eval-hub/eval-hub version:v0.3.0]]] workbenches:map[managementState:Removed releases:[map[name:Kubeflow Notebook Controller repoUrl:https://github.com/kubeflow/kubeflow version:1.10.0]] workbenchNamespace:rhods-notebooks]] conditions:[map[lastTransitionTime:2026-07-09T20:32:07Z message:Some components are not ready: dashboard reason:NotReady status:False type:Ready] map[lastTransitionTime:2026-07-09T20:31:03Z observedGeneration:2212 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-09T20:32:07Z message:Some components are not ready: dashboard reason:NotReady status:False type:ComponentsReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:AIPipelinesReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:0/1 deployments ready reason:DeploymentsNotReady status:False type:DashboardReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:FeastOperatorReady] map[lastTransitionTime:2026-07-09T20:28:23Z status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-09T20:28:23Z status:True type:KserveLLMInferenceServiceWideEPDependencies] map[lastTransitionTime:2026-07-09T20:32:07Z status:True type:KserveReady] map[lastTransitionTime:2026-07-09T20:32:07Z status:True type:KueueReady] map[lastTransitionTime:2026-07-09T20:32:07Z status:True type:MLflowOperatorReady] map[lastTransitionTime:2026-07-09T20:27:54Z message:MaaS gateway annotations and Authorino TLS are correctly configured reason:MaaSPrerequisitesMet status:True type:MaaSPrerequisitesAvailable] map[lastTransitionTime:2026-07-09T20:32:07Z status:True type:ModelControllerReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:ModelRegistryReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:MaaS platform manifests applied and maas-api deployment is available reason:Reconciled status:True type:ModelsAsServiceReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:OGXReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:RayReady] map[lastTransitionTime:2026-07-09T20:32:07Z status:True type:SparkOperatorReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainerReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:TrainingOperatorReady] map[lastTransitionTime:2026-07-09T20:32:07Z status:True type:TrustyAIReady] map[lastTransitionTime:2026-07-09T20:32:07Z message:Component ManagementState is set to Removed reason:Removed severity:Info status:False type:WorkbenchesReady]] observedGeneration:2212 phase:Not Ready release:map[name:OpenShift AI Self-Managed version:3.5.0-ea.2]]] to match expression <string>: .spec.components.aigateway.managementState == "Removed" --- FAIL: TestOdhOperator/components/group_1/aigateway (301.79s) === RUN TestOdhOperator/components/group_1/kserve/Validate_component_spec testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_1/kserve/Validate_component_spec (600.03s) === RUN TestOdhOperator/components/group_1/kserve === PAUSE TestOdhOperator/components/group_1/kserve === CONT TestOdhOperator/components/group_1/kserve test_context_test.go:223: Timed out after 600.008s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'default-kserve' of kind 'Kserve' to match condition {.spec.nim.managementState == "" []}. Expected <string>: map[apiVersion:components.platform.opendatahub.io/v1alpha1 kind:Kserve metadata:map[annotations:map[component.opendatahub.io/management-state:Managed platform.opendatahub.io/instance.generation:2214 platform.opendatahub.io/instance.name:default-dsc platform.opendatahub.io/instance.uid:191cdb15-01aa-4da4-9798-e6b508adbf9f platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-09T20:28:02Z finalizers:[platform.opendatahub.io/finalizer] generation:1 labels:map[platform.opendatahub.io/part-of:datasciencecluster] managedFields:[map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:finalizers:map[v:"platform.opendatahub.io/finalizer":map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"191cdb15-01aa-4da4-9798-e6b508adbf9f"}:map[]]] f:status:map[f:conditions:map[] f:observedGeneration:map[] f:phase:map[] f:releases:map[k:{"name":"KServe"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-inference-scheduler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"llm-d-workload-variant-autoscaler"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]] k:{"name":"vLLM"}:map[.:map[] f:name:map[] f:repoUrl:map[] f:version:map[]]]]] manager:kserve operation:Apply subresource:status time:2026-07-09T20:29:12Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[f:component.opendatahub.io/management-state:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[f:platform.opendatahub.io/part-of:map[]] f:ownerReferences:map[k:{"uid":"191cdb15-01aa-4da4-9798-e6b508adbf9f"}:map[]]] f:spec:map[f:modelsAsService:map[f:managementState:map[]] f:nim:map[] f:rawDeploymentServiceConfig:map[] f:wva:map[f:managementState:map[]]]] manager:datasciencecluster operation:Apply time:2026-07-09T20:33:12Z] map[apiVersion:components.platform.opendatahub.io/v1alpha1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"platform.opendatahub.io/finalizer":map[]]]] manager:manager operation:Update time:2026-07-09T20:28:02Z]] name:default-kserve ownerReferences:[map[apiVersion:datasciencecluster.opendatahub.io/v2 blockOwnerDeletion:true controller:true kind:DataScienceCluster name:default-dsc uid:191cdb15-01aa-4da4-9798-e6b508adbf9f]] resourceVersion:386848716 uid:d0766eb8-2776-49ac-ac5c-875a23e93557] spec:map[modelsAsService:map[managementState:Managed] nim:map[airGapped:false managementState:Managed] rawDeploymentServiceConfig:Headless wva:map[managementState:Removed]] status:map[conditions:[map[lastTransitionTime:2026-07-09T20:28:59Z status:True type:Ready] map[lastTransitionTime:2026-07-09T20:28:59Z observedGeneration:1 status:True type:ProvisioningSucceeded] map[lastTransitionTime:2026-07-09T20:29:12Z observedGeneration:1 status:True type:DeploymentsAvailable] map[lastTransitionTime:2026-07-09T20:28:02Z observedGeneration:1 status:True type:DependenciesAvailable] map[lastTransitionTime:2026-07-09T20:28:02Z observedGeneration:1 status:True type:KserveLLMInferenceServiceDependencies] map[lastTransitionTime:2026-07-09T20:28:02Z observedGeneration:1 status:True type:KserveLLMInferenceServiceWideEPDependencies]] observedGeneration:1 phase:Ready releases:[map[name:KServe repoUrl:https://github.com/kserve/kserve/ version:v0.17.0] map[name:vLLM repoUrl:https://github.com/vllm-project/vllm version:v0.18.0] map[name:llm-d-inference-scheduler repoUrl:https://github.com/llm-d/llm-d-inference-scheduler version:v0.7.1] map[name:llm-d-workload-variant-autoscaler repoUrl:https://github.com/llm-d/llm-d-workload-variant-autoscaler version:v0.6.0]]]] to match expression <string>: .spec.nim.managementState == "" --- FAIL: TestOdhOperator/components/group_1/kserve (682.80s) === RUN TestOdhOperator/components/group_1 --- FAIL: TestOdhOperator/components/group_1 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.16s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-09T20:46:04Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.19.18"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.ods-qe-psi-07.osp.rh-ods.com, certSecret=router-certs-default controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.008s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-09T20:46:56Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", }, <map[string]interface {} | len:6>{ "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-09T20:46:56Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", }, <map[string]interface {} | len:6>{ "type": <string>"Degraded", "lastTransitionTime": <string>"2026-07-09T20:46:56Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", }, <map[string]interface {} | len:6>{ "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-09T20:47:16Z", }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-09T20:47:16Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", "type": <string>"Ready", }, ], "phase": <string>"Active", }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "uid": <string>"bec64988-8cc2-4e74-aab7-83103a9e31ee", "creationTimestamp": <string>"2026-07-09T20:46:53Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"80d8c398-a822-4973-9cf9-81e5810d4f34\"}": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-09T20:46:53Z", }, <map[string]interface {} | len:7>{ "subresource": <string>"status", "time": <string>"2026-07-09T20:47:17Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"80d8c398-a822-4973-9cf9-81e5810d4f34", }, ], "resourceVersion": <string>"386865133", }, }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (511.41s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/09 20:39:03 TEST FAILURE DETECTED: group 1 2026/07/09 20:39:03 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 1) === 2026/07/09 20:39:04 === CLUSTER STATE === 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/09 20:39:04 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/09 20:39:04 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:39:04 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:39:04 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:39:04 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:39:04 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/09 20:39:04 === DEPLOYMENTS === 2026/07/09 20:39:04 Namespace: redhat-ods-applications 2026/07/09 20:39:04 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/09 20:39:04 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/09 20:39:04 Deployment maas-api: 1/1 (READY) 2026/07/09 20:39:04 Deployment maas-controller: 1/1 (READY) 2026/07/09 20:39:04 Deployment maas-postgres: 1/1 (READY) 2026/07/09 20:39:04 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/09 20:39:04 Deployment model-serving-api: 1/1 (READY) 2026/07/09 20:39:04 Deployment odh-model-controller: 1/1 (READY) 2026/07/09 20:39:04 Deployment postgres: 1/1 (READY) 2026/07/09 20:39:04 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/09 20:39:04 Namespace: kube-system 2026/07/09 20:39:04 === PODS === 2026/07/09 20:39:04 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-6mzvr: container manager restarts=2; redhat-ods-applications/trustyai-service-operator-controller-manager-6f7f88c874-q4r8g: container manager restarts=2 2026/07/09 20:39:04 === OPERATOR STATUS === 2026/07/09 20:39:04 Operator deployment: 3/3 ready 2026/07/09 20:39:04 === DSCI STATUS === 2026/07/09 20:39:04 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 20:39:04 === DSC STATUS === 2026/07/09 20:39:04 DSC default-dsc: 2026/07/09 20:39:04 Ready: True - 2026/07/09 20:39:04 ProvisioningSucceeded: True - 2026/07/09 20:39:04 ComponentsReady: True - 2026/07/09 20:39:04 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 KserveLLMInferenceServiceDependencies: True - 2026/07/09 20:39:04 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 20:39:04 KserveReady: True - 2026/07/09 20:39:04 KueueReady: True - 2026/07/09 20:39:04 MLflowOperatorReady: True - 2026/07/09 20:39:04 MaaSPrerequisitesAvailable: True - MaaS gateway annotations and Authorino TLS are correctly configured 2026/07/09 20:39:04 ModelControllerReady: True - 2026/07/09 20:39:04 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 ModelsAsServiceReady: True - MaaS platform manifests applied and maas-api deployment is available 2026/07/09 20:39:04 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 RayReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 TrainerReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 TrustyAIReady: True - 2026/07/09 20:39:04 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 20:39:04 === RECENT EVENTS === 2026/07/09 20:39:04 No recent events found in monitored namespaces 2026/07/09 20:39:04 === RESOURCE QUOTAS === 2026/07/09 20:39:04 No resource quotas found in monitored namespaces 2026/07/09 20:39:04 Classification for group 1: infrastructure/dsci-unhealthy (code=1009, confidence=medium) 2026/07/09 20:39:04 Evidence: DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"dsci-unhealthy","error_code":1009,"evidence":["DSCI default-dsci unhealthy: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False"],"confidence":"medium"} 2026/07/09 20:39:04 Diagnostics complete for failed test: group 1 controller_test.go:753: Stopping: group 1 test failed. 2026/07/09 20:39:04 Circuit breaker: failure classified as infrastructure/dsci-unhealthy (code=1009) 2026/07/09 20:39:04 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/09 20:42:26 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/09 20:54:35 TEST FAILURE DETECTED: group 4 2026/07/09 20:54:35 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/09 20:54:36 === CLUSTER STATE === 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/09 20:54:36 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/09 20:54:36 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:54:36 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:54:36 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:54:36 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:54:36 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/09 20:54:36 === DEPLOYMENTS === 2026/07/09 20:54:36 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 20:54:36 === PODS === 2026/07/09 20:54:36 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-qs4fn: container manager not ready 2026/07/09 20:54:36 === OPERATOR STATUS === 2026/07/09 20:54:36 Operator deployment: 3/3 ready 2026/07/09 20:54:36 === DSCI STATUS === 2026/07/09 20:54:36 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 20:54:36 === DSC STATUS === 2026/07/09 20:54:36 DSC default-dsc: 2026/07/09 20:54:36 Ready: True - 2026/07/09 20:54:36 ProvisioningSucceeded: True - 2026/07/09 20:54:36 ComponentsReady: True - 2026/07/09 20:54:36 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 KserveLLMInferenceServiceDependencies: True - 2026/07/09 20:54:36 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 20:54:36 KserveReady: True - 2026/07/09 20:54:36 KueueReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 ModelControllerReady: True - 2026/07/09 20:54:36 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 RayReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 TrainerReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 20:54:36 === RECENT EVENTS === 2026/07/09 20:54:36 20:53:45 INFO Pod/maas-controller-656b6867cb-qs4fn: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" 2026/07/09 20:54:36 20:53:45 INFO Pod/maas-controller-656b6867cb-qs4fn: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 778ms (778ms including waiting). Image size: 212345923 bytes. 2026/07/09 20:54:36 20:52:40 WARN Pod/maas-controller-656b6867cb-qs4fn: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-d588s); attempting to pull the image may not succeed. 2026/07/09 20:54:36 20:50:52 INFO Pod/maas-controller-656b6867cb-qs4fn: Created - Created container: manager 2026/07/09 20:54:36 20:50:52 INFO Pod/maas-controller-656b6867cb-qs4fn: Started - Started container manager 2026/07/09 20:54:36 20:50:52 INFO Pod/maas-controller-656b6867cb-qs4fn: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 806ms (806ms including waiting). Image size: 212345923 bytes. 2026/07/09 20:54:36 === RESOURCE QUOTAS === 2026/07/09 20:54:36 No resource quotas found in monitored namespaces 2026/07/09 20:54:36 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 20:54:36 Evidence: container maas-controller-656b6867cb-qs4fn/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-qs4fn_redhat-ods-applications(c31ad2de-34e2-4afa-9280-f1349ba44ce5) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-656b6867cb-qs4fn/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-qs4fn_redhat-ods-applications(c31ad2de-34e2-4afa-9280-f1349ba44ce5)"],"confidence":"medium"} 2026/07/09 20:54:36 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/09 20:54:36 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 20:54:36 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/09 20:55:47 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1747.34s) === RUN TestOdhOperator 2026/07/09 20:27:29 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/09 20:56:47 TEST FAILURE DETECTED: components 2026/07/09 20:56:47 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/09 20:56:48 === CLUSTER STATE === 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-master-0 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-master-1 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-master-2 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23450732Ki memory allocatable (of 12000m CPU, 24601708Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-worker-0-gc6qk 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23450728Ki memory allocatable (of 12000m CPU, 24601704Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23452748Ki memory allocatable (of 12000m CPU, 24603724Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-worker-0-jd4pm 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23452736Ki memory allocatable (of 12000m CPU, 24603712Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23452744Ki memory allocatable (of 12000m CPU, 24603720Ki memory capacity) 2026/07/09 20:56:48 Node: ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/09 20:56:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/09 20:56:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/09 20:56:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/09 20:56:48 Condition Ready: True - kubelet is posting ready status 2026/07/09 20:56:48 Resources: 11500m CPU, 23450720Ki memory allocatable (of 12000m CPU, 24601696Ki memory capacity) 2026/07/09 20:56:48 === DEPLOYMENTS === 2026/07/09 20:56:48 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/09 20:56:48 === PODS === 2026/07/09 20:56:48 Failed to collect pod data: redhat-ods-applications/maas-controller-656b6867cb-qs4fn: container manager not ready 2026/07/09 20:56:48 === OPERATOR STATUS === 2026/07/09 20:56:48 Operator deployment: 3/3 ready 2026/07/09 20:56:48 === DSCI STATUS === 2026/07/09 20:56:48 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/09 20:56:48 === DSC STATUS === 2026/07/09 20:56:48 DSC default-dsc: 2026/07/09 20:56:48 Ready: True - 2026/07/09 20:56:48 ProvisioningSucceeded: True - 2026/07/09 20:56:48 ComponentsReady: True - 2026/07/09 20:56:48 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 DashboardReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 KserveLLMInferenceServiceDependencies: True - 2026/07/09 20:56:48 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/09 20:56:48 KserveReady: True - 2026/07/09 20:56:48 KueueReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 ModelControllerReady: True - 2026/07/09 20:56:48 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 OGXReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 RayReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 TrainerReady: True - 2026/07/09 20:56:48 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/09 20:56:48 === RECENT EVENTS === 2026/07/09 20:56:48 20:56:28 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-z6kkx_bf326240-2a26-428e-9325-696c53e8d4ee became leader 2026/07/09 20:56:48 20:56:28 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-lwh58_360925a9-027e-4c9d-b8ed-3427de740536 became leader 2026/07/09 20:56:48 20:56:19 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-lj42m_41cbb365-c2e9-4980-88e0-e347f61b4ef0 became leader 2026/07/09 20:56:48 20:56:12 INFO Pod/llmisvc-controller-manager-586c85c465-lwh58: Created - Created container: manager 2026/07/09 20:56:48 20:56:12 INFO Pod/kserve-controller-manager-6c8cb86d7f-z6kkx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 984ms (984ms including waiting). Image size: 237680379 bytes. 2026/07/09 20:56:48 20:56:12 INFO Pod/llmisvc-controller-manager-586c85c465-lwh58: Started - Started container manager 2026/07/09 20:56:48 20:56:12 INFO Pod/llmisvc-controller-manager-586c85c465-lwh58: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 775ms (775ms including waiting). Image size: 235185383 bytes. 2026/07/09 20:56:48 20:56:12 INFO Pod/kserve-controller-manager-6c8cb86d7f-z6kkx: Started - Started container manager 2026/07/09 20:56:48 20:56:12 INFO Pod/kserve-controller-manager-6c8cb86d7f-z6kkx: Created - Created container: manager 2026/07/09 20:56:48 20:56:11 INFO Pod/llmisvc-controller-manager-586c85c465-lwh58: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-lwh58 to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/09 20:56:48 20:56:11 INFO Pod/kserve-controller-manager-6c8cb86d7f-z6kkx: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-z6kkx to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 20:56:11 INFO Pod/kserve-controller-manager-6c8cb86d7f-z6kkx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/09 20:56:48 20:56:11 INFO Pod/kserve-controller-manager-6c8cb86d7f-z6kkx: AddedInterface - Add eth0 [10.129.3.31/23] from ovn-kubernetes 2026/07/09 20:56:48 20:56:11 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-lwh58 2026/07/09 20:56:48 20:56:11 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/09 20:56:48 20:56:11 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-z6kkx 2026/07/09 20:56:48 20:56:11 INFO Pod/llmisvc-controller-manager-586c85c465-lwh58: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/09 20:56:48 20:56:11 INFO Pod/llmisvc-controller-manager-586c85c465-lwh58: AddedInterface - Add eth0 [10.131.0.145/23] from ovn-kubernetes 2026/07/09 20:56:48 20:56:11 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/09 20:56:48 20:56:00 INFO Pod/model-serving-api-78bbbb9bf8-4p4r5: Created - Created container: server 2026/07/09 20:56:48 20:56:00 INFO Pod/odh-model-controller-54bf759f64-lj42m: Created - Created container: manager 2026/07/09 20:56:48 20:56:00 INFO Pod/odh-model-controller-54bf759f64-lj42m: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 805ms (805ms including waiting). Image size: 238719956 bytes. 2026/07/09 20:56:48 20:56:00 INFO Pod/odh-model-controller-54bf759f64-lj42m: Started - Started container manager 2026/07/09 20:56:48 20:56:00 INFO Pod/model-serving-api-78bbbb9bf8-4p4r5: AddedInterface - Add eth0 [10.129.3.30/23] from ovn-kubernetes 2026/07/09 20:56:48 20:56:00 INFO Pod/model-serving-api-78bbbb9bf8-4p4r5: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/09 20:56:48 20:56:00 INFO Pod/model-serving-api-78bbbb9bf8-4p4r5: Started - Started container server 2026/07/09 20:56:48 20:55:59 INFO Pod/model-serving-api-78bbbb9bf8-4p4r5: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-4p4r5 to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 20:55:59 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/09 20:56:48 20:55:59 INFO Pod/odh-model-controller-54bf759f64-lj42m: AddedInterface - Add eth0 [10.129.3.29/23] from ovn-kubernetes 2026/07/09 20:56:48 20:55:59 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 20:56:48 20:55:59 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 20:56:48 20:55:59 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-4p4r5 2026/07/09 20:56:48 20:55:59 INFO Pod/odh-model-controller-54bf759f64-lj42m: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/09 20:56:48 20:55:58 INFO Pod/odh-model-controller-54bf759f64-lj42m: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-lj42m to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 20:55:58 WARN Pod/odh-model-controller-54bf759f64-lj42m: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 20:56:48 20:55:58 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/09 20:56:48 20:55:58 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-lj42m 2026/07/09 20:56:48 20:55:47 INFO Pod/odh-model-controller-54bf759f64-dhgww: Killing - Stopping container manager 2026/07/09 20:56:48 20:55:47 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: Killing - Stopping container manager 2026/07/09 20:56:48 20:55:47 INFO Pod/model-serving-api-78bbbb9bf8-sc75z: Killing - Stopping container server 2026/07/09 20:56:48 20:55:47 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: Killing - Stopping container manager 2026/07/09 20:56:48 20:55:47 WARN Endpoints/llmisvc-controller-manager-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/llmisvc-controller-manager-service: Operation cannot be fulfilled on endpoints "llmisvc-controller-manager-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/llmisvc-controller-manager-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 535a85e2-1246-4e75-9c7e-7e001646c65b, UID in object meta: 2026/07/09 20:56:48 20:55:26 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-586c85c465-v4slx_56b7036b-6b9b-42d9-93ab-77b44d98425d became leader 2026/07/09 20:56:48 20:55:23 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-6c8cb86d7f-fqs7b_fc850e8a-82cb-4f04-86c3-a35776df2559 became leader 2026/07/09 20:56:48 20:55:07 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: Created - Created container: manager 2026/07/09 20:56:48 20:55:07 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: Started - Started container manager 2026/07/09 20:56:48 20:55:07 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: Started - Started container manager 2026/07/09 20:56:48 20:55:07 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" in 804ms (804ms including waiting). Image size: 235185383 bytes. 2026/07/09 20:56:48 20:55:07 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" in 848ms (848ms including waiting). Image size: 237680379 bytes. 2026/07/09 20:56:48 20:55:07 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: Created - Created container: manager 2026/07/09 20:56:48 20:55:07 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-54bf759f64-dhgww_f96fc7da-7443-47c4-a477-869558d7fc06 became leader 2026/07/09 20:56:48 20:55:06 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: AddedInterface - Add eth0 [10.129.3.27/23] from ovn-kubernetes 2026/07/09 20:56:48 20:55:06 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-controller-rhel9@sha256:7f6f393635a7431d530256ef8ad48ebd35531166dbb361c6c0ff9fc37bd5dc2b" 2026/07/09 20:56:48 20:55:06 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: Pulling - Pulling image "registry.redhat.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:d3225eb3f0bfcb20a6d1dccd63403e58a810921680425ecae1b22d86a451417d" 2026/07/09 20:56:48 20:55:06 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: AddedInterface - Add eth0 [10.129.3.28/23] from ovn-kubernetes 2026/07/09 20:56:48 20:55:05 INFO Pod/llmisvc-controller-manager-586c85c465-v4slx: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-586c85c465-v4slx to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 20:55:05 INFO Pod/kserve-controller-manager-6c8cb86d7f-fqs7b: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-6c8cb86d7f-fqs7b to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 20:55:05 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-6c8cb86d7f from 0 to 1 2026/07/09 20:56:48 20:55:05 INFO ReplicaSet/llmisvc-controller-manager-586c85c465: SuccessfulCreate - Created pod: llmisvc-controller-manager-586c85c465-v4slx 2026/07/09 20:56:48 20:55:05 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-586c85c465 from 0 to 1 2026/07/09 20:56:48 20:55:05 INFO ReplicaSet/kserve-controller-manager-6c8cb86d7f: SuccessfulCreate - Created pod: kserve-controller-manager-6c8cb86d7f-fqs7b 2026/07/09 20:56:48 20:54:57 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-776d978f97-8jxld_13ef35c3-9b67-4eee-af2e-410e8089d1fb became leader 2026/07/09 20:56:48 20:54:50 INFO Pod/model-serving-api-78bbbb9bf8-sc75z: AddedInterface - Add eth0 [10.131.0.144/23] from ovn-kubernetes 2026/07/09 20:56:48 20:54:50 INFO Pod/model-serving-api-78bbbb9bf8-sc75z: Pulled - Container image "registry.redhat.io/rhoai/odh-model-serving-api-rhel9@sha256:6dd699e678c3f4d6e51138eea62f096f26472b5f9c1ce2757cf9e9e2f80371a4" already present on machine 2026/07/09 20:56:48 20:54:50 INFO Pod/model-serving-api-78bbbb9bf8-sc75z: Created - Created container: server 2026/07/09 20:56:48 20:54:50 INFO Pod/model-serving-api-78bbbb9bf8-sc75z: Started - Started container server 2026/07/09 20:56:48 20:54:49 INFO Pod/model-serving-api-78bbbb9bf8-sc75z: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-78bbbb9bf8-sc75z to ods-qe-psi-07-b4mqp-worker-0-wvlfv 2026/07/09 20:56:48 20:54:49 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-78bbbb9bf8 from 0 to 1 2026/07/09 20:56:48 20:54:49 INFO Pod/odh-model-controller-54bf759f64-dhgww: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" in 813ms (813ms including waiting). Image size: 238719956 bytes. 2026/07/09 20:56:48 20:54:49 INFO ReplicaSet/model-serving-api-78bbbb9bf8: SuccessfulCreate - Created pod: model-serving-api-78bbbb9bf8-sc75z 2026/07/09 20:56:48 20:54:49 WARN ReplicaSet/model-serving-api-78bbbb9bf8: FailedCreate - Error creating: pods "model-serving-api-78bbbb9bf8-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/09 20:56:48 20:54:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 20:56:48 20:54:49 INFO Pod/odh-model-controller-54bf759f64-dhgww: Started - Started container manager 2026/07/09 20:56:48 20:54:49 INFO Pod/odh-model-controller-54bf759f64-dhgww: Created - Created container: manager 2026/07/09 20:56:48 20:54:49 WARN Pod/model-serving-api-78bbbb9bf8-sc75z: FailedMount - MountVolume.SetUp failed for volume "tls-certs" : secret "model-serving-api-tls" not found 2026/07/09 20:56:48 20:54:48 INFO Pod/odh-model-controller-54bf759f64-dhgww: Pulling - Pulling image "registry.redhat.io/rhoai/odh-model-controller-rhel9@sha256:b423702816000ce0f075548e3a67de006f40ccf05c575a866cb62e2ac9711191" 2026/07/09 20:56:48 20:54:48 INFO Pod/odh-model-controller-54bf759f64-dhgww: AddedInterface - Add eth0 [10.129.3.26/23] from ovn-kubernetes 2026/07/09 20:56:48 20:54:47 INFO Pod/odh-model-controller-54bf759f64-dhgww: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-54bf759f64-dhgww to ods-qe-psi-07-b4mqp-worker-0-jcdqk 2026/07/09 20:56:48 20:54:47 WARN Pod/odh-model-controller-54bf759f64-dhgww: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/09 20:56:48 20:54:47 INFO ReplicaSet/odh-model-controller-54bf759f64: SuccessfulCreate - Created pod: odh-model-controller-54bf759f64-dhgww 2026/07/09 20:56:48 20:54:47 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-54bf759f64 from 0 to 1 2026/07/09 20:56:48 20:54:41 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-8jxld: Started - Started container manager 2026/07/09 20:56:48 20:54:41 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-8jxld: Pulled - Container image "registry.redhat.io/rhoai/odh-trainer-rhel9@sha256:904dd86da976fced2d3618cbe932baadaf94934280aab3a68f6cc9f04ee2efbd" already present on machine 2026/07/09 20:56:48 20:54:41 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-8jxld: AddedInterface - Add eth0 [10.130.3.193/23] from ovn-kubernetes 2026/07/09 20:56:48 20:54:41 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-8jxld: Created - Created container: manager 2026/07/09 20:56:48 20:54:40 INFO Pod/kubeflow-trainer-controller-manager-776d978f97-8jxld: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-776d978f97-8jxld to ods-qe-psi-07-b4mqp-worker-0-tx6lc 2026/07/09 20:56:48 20:54:40 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-776d978f97 from 0 to 1 2026/07/09 20:56:48 20:54:40 INFO ReplicaSet/kubeflow-trainer-controller-manager-776d978f97: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-776d978f97-8jxld 2026/07/09 20:56:48 20:54:37 INFO Pod/model-serving-api-78bbbb9bf8-sbpq8: Killing - Stopping container server 2026/07/09 20:56:48 20:54:37 INFO Pod/kserve-controller-manager-6c8cb86d7f-ztgjq: Killing - Stopping container manager 2026/07/09 20:56:48 20:54:37 INFO Pod/llmisvc-controller-manager-586c85c465-sf5vd: Killing - Stopping container manager 2026/07/09 20:56:48 20:54:37 INFO Pod/odh-model-controller-54bf759f64-5q5hh: Killing - Stopping container manager 2026/07/09 20:56:48 20:54:36 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - ServiceMonitor odh-model-controller-metrics-monitor was rejected due to invalid configuration: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/09 20:56:48 20:53:45 INFO Pod/maas-controller-656b6867cb-qs4fn: Pulling - Pulling image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" 2026/07/09 20:56:48 20:53:45 INFO Pod/maas-controller-656b6867cb-qs4fn: Pulled - Successfully pulled image "registry.redhat.io/rhoai/odh-maas-controller-rhel9@sha256:b8cca0d56d9ca5ab366cdc4042f13a908326e57b1b711c59e6ff101507058fb4" in 778ms (778ms including waiting). Image size: 212345923 bytes. 2026/07/09 20:56:48 20:52:40 WARN Pod/maas-controller-656b6867cb-qs4fn: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-d588s); attempting to pull the image may not succeed. 2026/07/09 20:56:48 === RESOURCE QUOTAS === 2026/07/09 20:56:48 No resource quotas found in monitored namespaces 2026/07/09 20:56:48 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/09 20:56:48 Evidence: container maas-controller-656b6867cb-qs4fn/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-qs4fn_redhat-ods-applications(c31ad2de-34e2-4afa-9280-f1349ba44ce5) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-656b6867cb-qs4fn/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-656b6867cb-qs4fn_redhat-ods-applications(c31ad2de-34e2-4afa-9280-f1349ba44ce5)"],"confidence":"medium"} 2026/07/09 20:56:48 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/09 20:56:48 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/09 20:56:48 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/09 20:56:49 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1760.56s)